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.ABSTRACT 

In this paper the author^ assesses 'the 
state-of-the-art on quality assessaent and aonltoring of ledical care 
and aakes recoaaendations for needed research in this area. Follo«j.Qg 
a brief ihtrbduction/ the content is presented in two sections. The 
first, providing a f raae of reference, covefs definitions; quality 
assessaent and prograa evaluation; relationship of qnality.and 
quantity; relationship of quality jind cost; strategies of carsJ 
structure, , process, and outcoae; aonltoring versus research; s^^/^^h-e 
uses, of butcoifes. The second section presents a catalog of ne«^®^ , 
research on assessing and aonltoring the quality vpf aedlcal care. Xhe 
research are^s covered are as follows: basic explorations and studies 
' of what constitutes quality, description of prevalent patterns and 
stitategies of pare, the epideaiology of quality, the relationship of 
structure to process or outcoae, developaent of basic tools fo^ 
assessaent, specification and testing of sys tea-design eleaents, 
coaparative studies of quality using different approached, further 
developaent of ^proaising current approaches, integrative leasur^s of 
quali^f applications to special areas, consuaer perspectives th-e 
consnaexJs role, quality assessaent and aonltoring as a social 
process, and effectiveness and the factors that influence it* An 
extensive bibliography is attached. (EH) 
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THIS boCOMENT HAS BEEN REPRO- 
DUCED EXACTLV AS lECElveO PROW 
''THE PERSON OR ORCAItiZA TlON ORIGIN* 
ATINCIT POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE* 
SENT OFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLICY 



National Cantar for Haalth Sarvlcaa Raaaarch 
flaaaarch Raport Sarlaa 

The Research Report Series is published by the Na- 
tional Center for Healt h. Services Research 
(NCHSR)-to provide significant research reporls 
'in their entirety. Research Reports are developed 
py the principal investigators who conduct the re- 
search, and are directed to selected users of health 
services research a6 part of a continuing NCHSR 
' effort to expedite the dissemination of i^w knoj/\- 
edge resulting from its project support. -' J 

v- 

Abatract ■ 

/ . ■ , 

The purpose/f this paper is to review, evaluate 
critically, an/ synthesize the literature oh quahty 
assessment and assurance, including the appro- 
priateness 6f use of service, in order to arrive at a 
cpgent, dcfcumenled, and authoritative assessment 
of. the st&telof-the-art. In addition to addressing 
quality Assessment as a research tool and quality 
assurance as an administrative tool, an attempt is 
■ - hiadp to provide an understanding of the 
epidemiology of qii^)i-as a prerequisite to the de- 
slgA of medical caJ&">8""™* ^"^ systems. Major 
compon«?nts of quajly/Vhich are discussed in- 
^ elude: definitions; quality assessment and prograrii 

evaluation; relationshijp-of quality and quantity; 
relationship of quality and c^t; strategies of care; 
. structure, pfbcess. and otitobme; monitoring ver- 
sus research; and the use* of outcomes. Rtcom- 
> ' mendations for further reseai^ch in the assessment 
an<| monitoring of the quality '6( medical cafe are 
.presented. 
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This NCHSR Research Report was written by Aycdis 
Donabcdian. M.D.. M.P.H .. Professor of .Medical 
Care Organization.- Department of Medita! Care 
Organization. School of Public Health, 1 he Uni- 
versity of Michigan. Ann Arbor. 1 his repoit. ati 
extended version oif a papei piqparcd foi the De- 
• partmcnt of Medicine and, Surgery of the Veterans 
Admim/trntion. is one product of the work cui'- 
rently/being supported by the" National Center for 
HeaUh .Services Research under grant number HS 
02aai. The final report from the work supported 
by NCHSR is not expected to be completed ^til 
June 1979. Dr. Donabcdian was also suppoMcti^by 
The University of Michigan, through sabbatical 
leave, during part of this study. 

Addiuonal copies^ of this report may be obtained 
6^,re|juest from the NCHSR Office of Scientific 
i-nd Technical Information. 3700 East-West 
lighway. room 7-44. Hy^sville. MD 20782 (tcl: 



,,117436-8970). Other current NCHSR publica- 
.^^Jtons are announced on the last .pages of this 
publication. . . ' ' 
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Prior to I965» the year in which Medicare became 
a reality, effoi'ts to improve the quality of medical 
care were involved mainly with the self-regulation 
activities of the medical profession, although 
sporadic research had been conducted in the areas 
of medical care process and patient outcome as far 
back as the mid-nineteenth century. Utilization rc- 
view was emphasized with the passage of Medicare 
d portended a new awareness of public interest 

flhe quality of care. Dr. Qpnabedian had theT)p- 
rtunity during the period of 1965-1967 to re- 
view and assess the st^te-of-the-art of quality as- 
sessment methodology and responded with several 
publications which have become classics in the 
Field. Since that time, there has been a period of 
significant, if not remarkable, growth in the body 
of knowledge encompassing" quality assessment 
and assurance. A notable development in this area 
was the implemehtation of] the Experimental Med- 
ical Care Review Organizatibn (EMCRO) program 
in Ipl by the Nation^il Center forjfealth Services 
flesearch. This program estaWjsficd the model foU 
lowed in the dbvelopment^f the Professional 
Star^dards Review Organization (PSRO) program 
and encountered many of the problems sub 
sequently experienced in that program. The 
EMCRO program, addressed problems of criteria 
development aijd evaluation, organizational pat- 
terns, development of assessment and assurance 
techniques, impact evaluation, anj!Kmany of the 
other enierging issbesiof the day. 

In October, 1972, Bublic Law 92-603 established 
formal PSRO review of medical services reim- 
bursed under th^ Social Security Act, and interest 
in all facets of quality research received added im- 
petus. The PSRO legislation has the potential for 
profound impact on the cost and quality of medi- 
cal services and the form of health services deliv- 
ery; however, serious concern has been voiced 
concerning the wisdom of its current mode of im- 
plementation, aspects of which have not been 
rigorously validated. ^ * • 



In the past decade, a considerable body of 
knowledge has been gathered which requires 
thoughtful review, evaluation, and synthesis in 
order to assess the present state-of-the-art and to 
allow meaningful projections of further research 
strategies. It is in this framework that Dr. Doitabe- 
dian suggests that "it is necessary from time to^ 
time to pause and take stock of what has been 
done, so that it may be clearly understood and 
future effort redirected." The synthesis which he 
provides is intended to mold together the re- 
search, operational, ^ind policy concerns of the 
health establishment with regard to the quality of 
medical care at a time when major changes in the 
financing ^r^d organization of health services are 
on the horizon. 

V 

Gerald Rosenthal, Ph.D. 
Director 

National Center for Health 
Services Research 

July 1978 
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Nol too long ago. the qualily of medical iaic was 
a matter Almost cxtlusi|ely in ihr piofessional 
domain. Any iniroduciion of ihv suhjcri in a niorc 
public setting was to be dor»c, i( al\all, gif'»Ki'ily, 
almost apologetically, surrounded a cloud oi 
caution designed to appease ihv wraihlul and con- 
temptuous professionut. How times have changed! 
"Qualhy" is now a term perhaps too easily bandit! 
about; and there is liftic hesiiance in proposing 
that quality can be measured, or thai it^can be en- 
forced as a matter of public and adminislrative 
policy. But this mood of almosi Mligereiu (onll- 
dence is perhaps premature, for ihere is much 
about the concept of qualily thai h elusive, unde- 
fined and untTKasured. Our^knowtedge^oLhow lo 
go about assuring quality is equally frail. 

The academician who seems to be pleading Jor 
inaction white proposing further research is a 
stock Tigu re of ridicule in our gallery of public 
fools. This is a role'I shall try^ to avoid. My thesis is 
that while some of us go about doing the best they 
can with what is known, the effortjto examine 
critically what is being done, and to findanew and 
better ways of doing it, should not be rcl*ced. 

This pap^r is offered as a modest contribution 
in this further exploration. It will present a catalog 
of needed research that is sufficiently organized to 
avoid. beiYig a mere haphazard listing of things. 
But first, a gpneral framework is needed thai 
explains and justifies the choice of research topics 
and their organization into a classification. The 
framework must also indicate what subjects are 
excjuded from consideration. For if reasonably 
strict limits are not set at the very beginning, the 
concept of quality has a tendency to expand, so 
that it embraces every evaluative statement about 
any aspect of the health care system whatever. 



Some claim that the concefM of quality is enriched 
w^en this happens. In my opinioni, this concept 
ca\i also become attenuated and less useful. 
Whether I am right or wrong, it is clear that I must 
circumscribe my subject rather narrowly if I am to 
complete this paper. For the same reason, the 
delineation of the frame of reference must also be 
sketchy, for a thorough specification of the 
fran»ework ts a formidable undertaking in itself. 
All tfiut is necessary, for now, i^hat we establish 
some common ground upon which to build. 
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Otflnltlont 

» * . , 

In some ways, my cjcfimtion ol qiKility will br 
uuiaually narrow. <^ualiiy is lakrn in Hignify a 
judflfmcnl on a rather limited segment ol the pet- 
formahce of some profcsftionat personnel. Primar- 
ily, our concern is with the services that are re- 
ceivcd and consumed personally and directly hy 
individuals, and with those protessim^ils who pro- 
vide such services. Thus, Tor Tnsiaruy. envircni- 
mental services and administrators Xf personal 
hAallh services are clearly u|ltsidr ourl scope; but 
the patient-care service oF physicianA. dentists, 
pharmacists, nurses, social^ workers, And cMher 
practitioners are clearly included. I belicW that it 
IS also reasclhable to include the services ()r^Tufes- 
sionals. such as pathologists and .diagn(>siTl 
radiologists, who generate the data that ate used 
in direct patient care. 

Traditionally, the activities of these several pro- 
fessionals are seen sebarate contributions. 
Hence, one speaks of the qiLlity of physician care, 
nursing care, social work, and\so on. In this paper, 
I shall foni>i/yiis rraditi<^ by focusing almost ex- 
clusively on *e services of physicians, but this is 
^ not by prcferibnpe. It is a necessary concession, in 
part, to the VelStively less deWoped state of qual- 
ity assessment outside medicine;*but. mainly, it is a 
reflection of my ignorance of much of the work in 
assessing the quality of performance in the other 
profeftions. In fact, an essential component of the 
*toncept of quality is the interrelatedness of the 
'^contribution^ of the several professionals in the 
management of .a patient's illness and health. A 
majoV item on the research agenda rs, therefore, 
the development of **imegrative" mea^res of qual- 
ity that ta^e into account this hiierrelaiedness. 

^Another way in which the concept of quality can 
^ be expande<;^ is by acceptiiig a definition of health 
that is considerably broader than the traditional 
- emphasis on physical-physiological function. I 
shall include psychological and social yell-being as 
necessary components of health, but only to 
€ gf hat responsibility for the^ can be a^ro- 
. lERXC "Overtaken b>^rt|t ph|ysician, und^a rea- 



sonably lil)eiul definition of his ryle. It is not use- 
ful to dcfineMhe lesponsibilitv/^Af a health care 
practitioner, lor any aspecj ^f hegjtth. so that it 
goes l>eyond hi.\>acr^ly and pvi)ft?ssionally 
sanctioned role, ut^the'lmTnimeiiHitities actually or 
poieiiiially uj>der his control. 

Quality atMstrMnt and program avaluatlon 

Program ev^/luation differs from quality a^sc^s- 
meni by being more inclusive. It deals jvith ac- 
tivities of health practitioners in addition to direct 
patient care. It also includes ihe performance of 
professionals and nonprofessionals who prt)vide 
) no dirdpt patient care. As a result, there is more to 
-^-progrrfm evaluation than quality assessment; al- 
though quality assessment is part of program 
evaluation, and in many inst^||K:es its most img^- 
tant pan. > 

In addition to this difference in extent or in- 
clusivity, there are differences that flow from the 
level of aggregation at which the assessment is 
performed. Program evaluation deals at the 
^Kg'^cgatc level with the manner in which a com- 
plex formal brganization functions, and whether it 
meets its socially legitimate goals or some specified 
needs or wants of its clients. In so far as the health 
care function is concerned, program 'bvaluation 
places greit^r emphasis on access to care, on other 
aspects oi resource allocation, on cost and effi- 
ciency, and on , overall impact on a clientele, a 
community or a population. By contrast, it has 
been customary for quality assessment to focus on 
the appropriateness of care at the level of personal 
interaction between' individual patients and prac- 
titioners, as judged by the proper application of 
science to the needs that the patient and the 
practitioner have jointly defined. However, as 
health care becomes more^ formally organized, and 
its financing collectivized/ there is increasing pres- 
sure to intr6duce into the assessment of quality at 
the individual level concerns that were previously 
confined to the collective level. of analysis and 
assessment, This hals raised serious ethical prob- 
lems for the practitioner wha is now caught be- 
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iwrrn ihr iw<i niilUn>nr» of rmpciimilMliiy (ow*iicU 
Ihr imliviiiilMl |mliriM <iim{ thr collrilivitv It Ilia 
iiU(» blwnrd l«rmril> t ituirr iliiiiin liniii hriwrni 
pniKr4m rvitluutton and ijualiiv 4Mr<«*mnu ili.ii 
hiivr IU)W Ik'ioiiir iihimIv clillnriMrn in rinplM)*iH 
4iicl ill ihr (IriaiU ot tin luiil.ilioti. Sonic* iiiiplii.t 
lioim ol ihln hint l ing ami ovnUip will hv lUi ilinl 
in )iiih»r(pirnl ^n licnii oj ihin piiprt 



Quality and quantity 

The civrrl.ip tiriwrrii qrinriTnv und qnnlttv tntr 
ninimt Ihiinigli winiii pioKi.inr rv.ilnaiion ton) 
cpiilli^l^^rHJOiUMn How ililncMi li iMxri , 
cpi;inlit;itivc* .iclnpuo in .1 nnrss.iiv pUMundoion 
lor cpialiiv t aicv I hin nu-.Mi% ili.o .u rc ^is 10 i aw .nid 
iii!iiiniciriil tTx:^cd sn viir .iir lrxitint.Hr rlrnirnis 
in ii^Jir'ijinK ihr ipi.diiv «d c linn pi .n liiionri ii .nis- 
anion .iV wril as ol ihr pinxiain as a wholr. Insnf 
fu irin .McrSH .0)ii nsr nrrd sprual and lonnnninK 
anrniion hrcaiinr so niaiiN pinK»'»"^'* ohsrs 
Hivrh^piriMcnpird wilh c nihieiK niili/aiion and 
f nitiiiK (osis, rvrn ihniif<h !hr\ nia\ pa\ lip vri vicr 
to ihr impoiiamr ol rliininaiiiiK iiisnHuirni lair. 

Kxt r^sivr usr ol sri vie r iiMialK inr.uis pool tpial 
it\ loi a v.i/irU ol iraMniN. hen onr. il implirs a 
lark ol ski/in ihr (ondmi ol palirnl rair: an in- 
ahiliiv 10 MHcMrrd from sirp lo slrp .ilon>? ihr niosl 
cliirt l--|wnh, scire lin>? ihr pinisrK nrrrssai v and 
siiincirnl prorrdmrs lo anivr al a diagnosis anch 
to in?nituir tiraimrni. I his nnxlrl ol logical and^ 
parsimonious piogrrssion has iiadiiionallv brrn 
ihr hallmark of viMuosii> in inrdical <air, and 
many still hold il as ihr idral. .ilihouKh iiuk h ol 
rurrrni arirptahlr pradicr ma\ havr drparird 
from il. Notr, howrvrr, thai this is an idral ol 
minimum rrdundaiuN. hul not iircrssarily of 
nitiiinium losi. Ii'is possihir, lor rxamplr. that 
usinK"* rurrrni trthnoloKV, a sirairgy ol rxirnsivr, 
almost indisrrinunait'. initial irsiiiiK will Irad to an 
rarlirr diagnosis, with K»wrr likriihood of rrror, at 
lowrr avrragr rosi, at Irasi in soinr siluations. If 
so, ihr ( rilrrion of lowrsi tost rould hr in tonllirt 
with the rriicrion of Irast rcduiidanc y as rlrnirnis 
in the drnnilion of cjualitv VVr shall irturn to ihis^ 
int err sling confrontation. 

In other ways, the ideniif liation of poor qualiiy 
with excessive use is less prohlrmaiir. Medical 
procedures, ihcmgh inirnded 10 hr henrfuial, are 
not wilhoiii risk which, in some cases, is consitlcr- 
^ble. Il is reasonable lo Assert that the concept 4)f 
quality includes ihe criterion that in no case should 
the benefits expected from any procedure be less 
lhan the risks it poses. If so. a problem in assess- 
moot is the measuremeiii of risks and benefiis in^i 
manner if>al permits a comparison. 

Still another undesirable consequence of exces- 
sive use is that by allowing some to have loo much, 
there is'less available t» others. I here is an obvious 



Hounr*, whij h ii 
nrl II tn jfinnloi 
r, llir ^aitri la o 
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miial ini»alhHalion of irHonnr* 
(pialilv al Ihr aKK'<*K»>^'<'^«^' ^ 
%cc thai, in lhi» iniiam r, llir nn«^t <»' 
irdnmlanir al ihr Irvrl ol i/idividiial paiiriU taic 
and al thr Millritivr Irvrl ^« all loi ihr Mmr lir- 
havioi, indicating a haimonicni?! MMinnrncr ol 
cpialiu aHHrj^impil'.ind pioniam rvalualioii. 

Onr ci|||^<iiW«tlr (Imt any rvahiaiivr nUiriiicni 
.ihoni lhe*uaiihlv ol tair.'bi'vond a inrrr drmrip- 
tiint (d it, in a jndMinrnt alntUI MMiie anpni ol qiiah 
it\ hn ihat 1 ravin, luili/aiion irvirw and qiialily 
.issr^smrni air uirxtiiiahl> intritwinrd. and will 
hr so < oiisitlrird ni tins papri 



Quality and coat 

\Vr havr ahrad\ had intiinaii<ms that tost is im* 
pliiatr<i i\\ thr roiurpl oT qiialiiN in a mannrt that 
is likriy to i ausr ns piohlrn»s I am nol irlri rinK 
MinpU thr lad that thr rinpiii<al rrlaiioii?*hip?* 
•hrtwrrn quality and cost arr rssrnlially nn-' 
rxplorrd. I am roiirrrnrd with thv comepmal 
lonnrtticm.H in thr drfiliitimi ol cpiality itsril. 

I hr basic rcHisidri .itions that link (pialttv to costs 
;irr rssrntially ihr samr that (onnrcl (piality lo 
quantity: monrtaiy cost, hrnrfil to hralth. and risk 
to hralth. 

I hrrr arr two ways in whiih ihr monetary costs 
ol inputs into c .irr can im leasr withoul any in- 
crrasr in the <pialit\ ol «arr. First, thr rlcjnems of 
service that go into patirnt carr can hr provided 
inefflrirnlly^^^^us. hospital carr will hr (osihrr 
9han -nrcrss^y if hJ>spital beds air eniply,,or ihe 
hospital IS impro|H'rly managed; 01 if ph\ sic inns 
do thr work of nurses and ihr latter thr work of 
titles. Srcond, thr elrmeins of carr i be com- 
bined and srqurnced in a maimer thai dors not 
rralizr thrir ful| potential to improve hralth. 

I hr luiidamriital attributr of ihesr drficienc ies 
in the organization, production and applicatym of 
care is that thrrr arr addrd costs withotft rithrr 
addrd brnrlits or addrd risks. Can this br con- 
st rurd as poor cjuahty of care?- I havr alrrady ai- 
gurd that at the collective lr^?rl ol analysis the an- 
swer could legitimately In- in thr alllrmative Ix'- 
causr, when rrsourcrs aiV scarce, wasieful produc- 
tion and implementation of rait reduces the po- 
irntial henents of care in thr aggreg|ie.>I he an- 
swer could also be in the affirmaiive l^the-level of 
the individual^practiiione'r-client inieraciion if re- 
sponsibility for (nefficiency falls, at least in part, 
on the practitioner. Po justify this conclusion we 
can draw on two arguments that I have already 
used. First, inefficient use of resources^suggcsts 
lack of skill or judgment' in the conduct of care: it 
is a manifestation of "logical redunxlV^^y ' Sec- 
ondly, it is a hiisalliKation of retources, no less at . 
the individual level than in the aggregate. The" ( 
individual pays more than he ought to for care 

in : ^ 



tltbf r immedlalcly ind directly, or in ihr IXiuirr ma 
the conttquencfi of profrtttn irtcmdrnry rvr«iii. 
•Ily work their way b(M:k to burden the bcitvidiiMl 
who mutt, ultimately, foot i|ie bill. ^ J/ 

Not everyone will agree that waiirfut tart i» l air 
of poor quUJIy. It ii nut important lu have agree* 
on thA. What is important in that, In rvalua- 
Ikmt of quality, the elemrnt of waiirtulnriii i>r 
dearly icientlfied and aMe»ied. Oiut thii ii <l^nr. 

decision^ could be madr riihrr lo l^rep it 
sqMirate or to merge it into an overall meanMie ol 
quality. 

There i i no dmgreen>em thiii the hstbtirr nt. 

health beneflii and rinkn In ai ihe tore ol qualiu* 
atiessmehi. To do no lurni is ihe tun%\ luillowed ni 
precepts in ihe clinicul iradidon No elenietii ol 
care should be used if i{% risis exceed in expcc led 
benefits. Similarly, ihe combinaiion ;ind i^eip^m 
jng of elemenis should leidi/e ihe liunesi 
relative lo the risks incurred. I jihal] return e6 ihis 
last stipulation when I deal with the siuitegie!i of 
care. 

If quality is measured in terms of iutiud wt 
expected benefils lo heullh from a spec ified ( oursr 

\of action, ihe relationship between cosIh a\h\ <|uyl- 
ity depends on the manner in whic li the liiMorji I 
described above enier into nire. I he elimin^iiion 
or reduclion of wastefulness will allow i\% to have 
the same quality at lower cost, or higher quality ai 
the .wme 'cost. This also holds to the extent that 
current care includes components that rarry un- 

_^necessary risk. But the presence of suth compo- 
nents adds another, more sinister, aspect to the 

Eire, h is possible for care to actually deterior^ 
io quality as costs increase above a rertain 
t. If the added services include a large enough 
component of high-risk, low-benefit items. 

Let us assume that the elements of rare afe 
produced and applied as efficiently as possible, 
and that only elements of care are used that have a 
demonstrable benefit. The relationship between 
qualitf-and costs, as more and more elements ty[ 
care are added, is empirically undetermined. One 
can. however, reasonably assume that the most 
bene^cial elements are likely to be used /irst and 
thai, as care becomes increasingly elaborate 
through the addition of more and more elements. 
^ the relatife gains in be^nefits and quality become 
smaller and smaUrfTln other words, there are 
diminishing returns and. in the end. very smafl 
advantage to beUerived from further enrichments 
in care or additional quantities of it. If we'accept 
this picture, the Cntial question becomes: At what 
level of inputs into care should the standard of 
quality be set? If individual patients paid all the 
costs and reaped all the bericfits the problem 
woul^be easily solved By saying that the patient 
and his physician cap joiiptly decide at what point 
ill l." 1 cosu are not worth the added benefits. I * 
*^ERiC»at patienu and physicians are capable of 



making «ut h judgments, using the relatively In- 
tomplrie information available lo ihetti. and when 
tbe alternativei are rather clearly demarcatetl But 
in Older to make a nM>re precise deterittinHtion. 
the data cm c osir and bettetlts must be tnuch^nrlore 
accurate, and reducible to a common utilt of 
i omparlnin—lor example, dollars. 

The »olution we have desi rit>ed is clearly in the 
mainMiram ol the clinical tradiilcm liecfuse It rec- 
oKiii/e* that patients differ, not only in the stu^ctly 
"medical" lectures id their illness, but also i^ihe 
valuation* ihe> plaie on the co«ts of larr ,»iid )u 
cxpcc^t^U bf^nefit*. ^larrifHl ttr an rxtfrr^ liin- 
(ormtdaiton would sup^Miri the \fui (vrhtig man^ 
(linician^ have that the quaHiy of care resi.^im tm. 
matiy indiviiiiial varilitionti to pertnil a general 
Mandard. What prevcits it from going quite that 
lat IS the large area of agreement among patients 
and prai titioners on the valuations plac ed on costs 
and benefits. 

Under the soluticA proposed «b<ive.'the quality 
m of care at the collective level would lie the sum 
total <if the quality <d care determined case by c ase. 
But ihisn:ttITrTusioii could be invafid if the costs 
were ( oilec tiviated, for example through health in^ 
uirancr;Sf the l)«4iefits of heahh ^arr were shared 
In more than the Vec ipierU; and if there were some 
MM ially legitimate reason for valuing the health of 
some, for example children, more than the health 
ol othei's, for example the aged. To the extent that 
the rollective interest differs from (he individual 
interest, and the health ptaf titioner is made the 
custmlian of botbs serioi/s strains are introduced 
into the patient-practitionc i relationship, and the 
definition of quality fjkisVs a'nioral dilemma. 
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Paiirnt t nfr in a planned activity which involves 
rlir choiic of specific elements oKare from a po- 
tentially large pool of such elements, and their 
pr«>per srquencmg in order to achieve specified 
diagnostic and iherapeuyc objectives, A plan of ac- 
tion, as well as the patw^n of actions it generates, 
may be referredtio as a strategy. A very simple 
example can be Issenibled from the work of sev- 
eral investigators who hVve dealt with the choice of 
strategies for the management of acute pharyngitis 
V in children and adulis. * 

Acceding to Brook, the development of process 
criteria in cooperation with infectious disease ex- 
perts resulted in the following^recomny^ndafions 
for the treatment of adults with sore throats: 

"If Wstory and physical indicate evidence of a sore 
throat, ^ throat culture should be dohe. If the culture^ 
IS pc)sitive. the patient should make a repeslf 
within 24 hours and t>e given a shot of procaine^ 
(short acting) penicilliA; he should then wait 50 min- 
utes in the office so that he canlbe promptly treated if 
an anaphylactic reacobn occurs. After another 24 
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hiiurt. iHf iwMPoi *h..iil.l inuiti, tl if- jH^ttu illtn 
ptitkllllii " ' I 

•irtilrn»*«actl4rnl..'iii \Ur 4iiih.»tiis .»» r»|>«^'y" 
tvurmciil "uinHiMl^iialiiv .»< i^i* " I ..mpkn/n 
ill brolMiM' infrW whrlliiMN* Mmil^i MMirgv l«. 

Im<i. opHiH*.! t»v innipai.iiR ilnrr In | int.^l 

WiulrKifi '<•• <*«r iiMiuKcturtu .»« m.ic tin. mi hi 
•tri «oii% who .h. 110/ Ipvr .1 lu^toi \ ..I |k ni. iHii. 4I 
IrlgV oia.l .hrmii..!.. Irvn I lit ih.rr .•hri i.....v.-, 
air 

• A ihr oinrniU 'irM»nunr...U«l ^i.iifKs, • l"""- 

Ihr ihioui ol jII iMiiri.i-. .im<I •<• I'r.ti >miIi |.riiK iI 
Uii iho^r v»lm»r (ullulr^ .lU |tu»iiiv, li.i Ki'»>i|. A 
^lir|tiiMiHi I. 

H iirji 4II jwiirni* wiili pi iiii iIImi i^ihk •< 

t\VtU\ i iilniir. ^ 
C urtilin (giniir noi iir.ii .iin |».iiirnt 

I oM.tn|>li<.Ur iiMllrM Imili. i . ih. xr sli .lU-Kirx .nr 
irMnI loi rpulnmt vtimi* . lulrmi. oMiiiinur n1 
Ihr t}i%rA%c. loi 01.1I vciMM |Mi.nUi.il iKMiitiUm, 
.111.1 I.. I clillrimi rxpr.ir.l i..i. -. ..I |H.Miivr ihn..ii 
tiihiiir in ihr |M»puJ.iln»ii n poilniK l«'i < 

K..ri>llj h.i» m.i.lr llir »po. ili. .iiu.ii .»« *it.ilrKir"» 
loi Ihr maiKiKrinrni ..I .mil.- ph.uM.Khi^ rvni 
iii.Mr K.inplrx hv inno<l«i.iMK -'^ •> ^"'P '"'' V 
ih:ii ..n ihr h.m^ ol .lum.il ri.i.linKV .LihuK < -i^r^ 
a* (I) inoM pn.h.iinv ..mimmI I»\ K""'P 
hriiM.lviu Mrrpl.Hi.MiM. ('J) ii»'»M piohahh iioi^o 
<.u«r<l. .111(1 (.H)'.pirMi..ti.ihlr .1^ lo rtn.loKV. VVilh 1 
fiirthrr *iiIh1ivimoii M ..i^ri iiiio .i.lull^ .'ii.l thil- 
.licn MX lairgiiiiri ol p.iiiriiiN .ii.- i.U niilir.l: ihirr 

hy CI1..I0KV <^"» ''^ " " 

iM.Mihlr (1) 10 irr.ii ail i.iHr> wiilioiii piioi cuiiur- 

iiig. (2) i.> .uliiiif .111.1 ilirii ii. Mi. .iikI (^) 

irriii ill all i.» iir.iiiiinii wiih priii. illtn. n 'an 

Ih- cilhri ..lai ..) p.irri.l.i a! 1 hus. ihrir i% .1 

matrix .>l S«> {x.^nihlr . omhiiiaiioii%. aii.l lor rai h 11 

is iMmiblr 10 U M llir i .msnpu n. rs^ 

F.lalH.rair as anythis srrins 1.. Ih-. ih^rxprneiur.i 

clini.ian will ivi/*nm/c, ii as an .wrrsiinpliluati.Mi 

of a'siiuaii.Mi ihai is ii«-lf rrlaiivi lv siniplr, Morr 

empiric sprc iruaiion ..I ■ Mrau-Ku s rr.piirr. ilir 

tonsiriirii..n nl raihrr rlal..»rau- proi.uols. al- 

' aoriihms ..r (lerMmu^irrs S.imr rrlcrni.r 1.) this 

work will be rnacU- in a sUbscqumi scrM..n All I 

want to do .l.> lu.w is ro lav .l.)wn ilir l.>uiidaiion 

for asserting that thr .Icwripti.iii an.l asscssinrni 

of the rlrmrnts ol rarc."..nc ai a time, misses the 

' design, the raiionaJe an.l ilic impluati.>ns ol the 

strategy as a yvht.lr. inrlu.ling ihr . ..nsr.piciurs .)f 

taking as well as n.)i taking icriain anions. In iin 

opinion, the verv essence of quality, that elusive 

but all-imp<»rtaiit ingrc.li^nt that we .all clinical 

iudgmeiH. resides in thr choice of the m..st appr.>- 

priate strategy fw the' management .)f anv given 

situation. I also believe that wr n.>w have ihc 

peceswry tools for sTpecifying and testing such 
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»ii4lrK»r». wlit.h M1P4I1* Ihrti ihr myMrrlr* uf tlilll- 
,al ni.lK»»rii« air ainriMblr u. virhlHtK ihrlr d»rk. 
r«l *c.irl» ntr*r hm.Is uir .IrcUUih *niily»U JMMI 
ilip aiialv»i» ol rllr.ilvrurM uml lun lH-nrflu. 
hill alih.uigh ihr*r iih>U rtir rtViilUblr. lhr>4Um 
that air iirr.lr.l (01 ihrH pircUr rtppUi«lU.M »rr 
grnrMllv Ukhig. Wr ncr«l m«»rr m.mtUr li»h>r-r 
maiinri Mnu thr .H.unriHC .»! \\U\c** 
.Imi.al an.l lalH.Mi..iv lindiiiK» i" «»»<Hiatl.>n wMh 
«uih ilhir**. al>lini iKi- in.»nrlrtiy i.»M» «»«hcr 
a»s.HMir.l wiih .Uagn..»tk prtfcrdurr*. ur 
vaM.m. ,r.|uri»r. .»l thnu. in ciirclly Idrnllly Hg 
,l(„r** whrn ii rmsu an.l rii.Mirously mining 
rxiMiiig ninrs».,all..iil itif rtiK^. buntrtt* aiwl 
.nonriai> .oMs ..I alirinaiivr ihrmpirt »% com- 
paicl to ilirii ...nnil.uli..ii» to hrrtlih; aboyl ihr 
ulativr val.iaii.ms 1.. Ik- plaicd .>n van.uis m«ii- 
ilrMati.nn .il hralili .h iH hralili «. that ihrsr «an 
l.r .i.l.lr.l lip nii.i a wrigliir.l sum thai aciuralcly 
irllriis ihrii i.»ial iinpa.i; and. where ro«|.lKnirlli 
uHnparis.His air 1.. Ik- ma.lr. a ...mm.m uiiH of 
mras.iirKirni is iirrdr.Uhai prrmiis a nMnpttri»oi»/ 
Kvm whrn «ll thr InRn iiiaii.ni nrrdrd l» not 
.ivailablr. manv "insight* nu.. 1 liniial Ju.lgmrnt can 
|,r gainr.l bv making usr .d what is already kn.hvn , 
siipplrmente.1 hv tjir ..pini.ms and valuations .>l 
thosr wh.mr pr.uti.r is nn.lfr study, an.l of iheir 

paiirnis. . 

()hvi.»usly. a .lrs<ripiion ..f ihr lcchni.|ues which 
wr hav^ mrnii.mcl i« iK-yond thr scope?.)! this 
pa|>rr l.usic.l has provithc.l a rras.»nably simple 
rxp.isiti.Hi ..f thr clinical applicaii.ms .d driision I 
.,n!dvsis « A ret rill paprr by McNeil et al. may 
srrvr as a tpiick inir.Hlfiction » I'hc nuire ambiiiom 
ira.Irr wi(h an apiimcir lor quantitative method* 11 
, rrfrrrcd to a in..rr rigor.uis exposiii.m by Raiffa. 
Klarnwn ha» publishe.l a brief account .>f cosi- 
rflr. n'vrnrss in.l . osi-l)enefii analysis as applied to 
mr.lical irchn.»l..gy . an.l provided an excclleni 
bibli.>graphv ' Kurt^her discussi.m ol the basic - 
meih.Hls rrfrrre.l i.)\»bol|Band a largr number of 
applicati..n» t*» the ass^MrfTent of strategies tlT^ 
snrgUal carr. iifcc to be foiKd in an excellent book 
edited bv Bunl(rr el al.Vl>Ar<^fer to .ither wolrk ^ 
in a subsequent sectiyfToT this paper; but 6nlr*f 
surface cap be skHrimed. kince this is an area 
cnrrently iin.ler intensive investigation. .We are. at 
last, experiencing a major advance in our under- 
sunding uf key elements in the quality of 
is a vcrv phlegmatic person indeed who will t)ol be 
siirfed as he first sees, as it were frcim a moun- 
laintop. this new an.l enchanting landscape! ^ 



Structure. proc»«t, outcome 

In ..ur frame .)f reference, when a judgmi^ni is 
made .m the quality of care it is taker^ to be by 
dcnniii.m. a judgment primarily on what profes- 
sionals do. and h.)w they behave, as they interact 



■ dirc:<;tly wU^ patients. Hence, it is the process 
of cateflhaf is,' ultimately: the object of quality as-^ 
sessmeht. 'Quality is AefixH^d as the degree of con-^ 
formance to, or deviation fcom, normative be- 
havior. J[n this formulation, both structural attrib- 
\ utes arfd outcome^ are'indirect means erf obtt^ining • 
- information about the hormativeness of process. 
• ^*The rather secondary role assig'ned to the as- 
- -sessment of stjucHire.is nof Hkely^to; be 'Seriously * 
^chall^hged. Accordiriglyrin-^rderTo jiia^ie niy task 
'm^anageable^l shall hlave Httle' to^say about struc- 
ture *in^ thiipaper: aUh<iugh it will;not be entirely 
eitfluded. The situation is ecrtfreiy^ different with 
resgectjguo.vtfooi^s,. slrtce,- according to a large 
^body of opinion, including "that of many leveling 
*^perts in quality assessment, it is- the outcome 6f ' 
care that is ihe. primary objecf^jl concern, and 
process onry a means to 4he attainment of out- - 
. co/iJe. It is.withMiffidenc?, and with some apology, 
'that a^.meth'od, based on process^ assessment c^a ' 
vpntpre iptbihis hostile environment; vdiereas it - 
a proud badge d£; honcir, assurihg ^Imdst ins'tant^ 
^attention and respect, to say, tlfat a method^, is ' 
"6utcome-opented/J-;Some have"^ ho hesitation to 
^ven distort reality, relabeling process elements as 
oijlcomesv in ordet- to avoid . th^ ' obloquy that 
attaches to process alic| to Basjc in the approbatian 
that .outcomes confer. . Of course, this picture 
'bverdra^.n, but not by much! ' > 

It \is not true that outcomes ^re a more valid 
, measure of, quality than is process, a§ it is fashion- 
H^ble to say. It is true that process mrieasu res are 
valid indicatory of quality only to the extent that 
they relate to relevant outcomes. But it is equally 
' true that outcomes are valid ineasures of quality 
pnly to the extent that they relate to the antece- 
d€?nt process of care. Fundamentally/validity de- 
pends on the strength of the relationship between 
process and outcome, and on our understanding 
of that iRglationship. If that vital link is weak; or in 
doubt, neither process nor outcome can be used to 
asses^ quality; the validity of both is attenuated,^ 
and to an equal degree. This means that, in this 
instance, we do not know what constitutes qu4lity. 
Further Vresearch is needed; the link between 
process and outcome must be investigated. In such 
an investigation outcomes are the^only reasonable 
criterion. There can be'rti^ disagreenjent about 
that, provided the proper outcbmes are selected 
and appropriately measured. 



ing knowledge tif that relationship'is used to obf 
t^i n information about th^' behavior of profes- 
sional personnel or of the larger system. - 

But, do^s monftoring have a subsidiary lysearcJf 
function? Is monitoring analbgous to clinical 
medicine, where a physician learns how to manage 
cases by observing the outcomes of care? I think 
the answer is both *|^es" and *-*No."'It is *T^s': ifT^ 
restri^tfd sense: if th&. occurrence of ugelE^ected* 
good or bad outcomes Ipa^saXo '^ review of process' 
• in the iight 'or cu\j.ef)tly .kn^wn relSlioxiships *be- 
tween the two; and if, iW the' event current knowl- 
edge does not provjAe^n answer, qwtpstion«- are* 
« i-aised about that Jcnqwledge so aV? suggest fur- 
tKer research. Jh|^ answer fs alhioSt always "N^^" if 
we expect tbe monitoViri^^echanism itself .to\be 
tha^ Uirthcr Fesearch;* T^&*tablfshmem of- jjew,. 
^inkages'b^tween process ^3 outcome-can qhly be^ 
achieAred,.v\^th;any^erta^^ .through carefully 
controll^ and meticulously ^condUcted> clinical 
trials.vit is unreaionabfe and dangehnisj'O' expect 
th^Ueyery PSRO, err its analogue, can function as a 
research agency ^\hat tells us what good mejticine 
is. One might with e^al reason assert, ^s has been 
done tn the^past, thal ihe best tes\bf t.he useful- 
ness of^a drug is the sum oi[ the*judgmehts of indi- 
vidual physicians as they observe i^ts effects on the 
' management of their individual patients, 



The uses of <iutcome8 



Monttorihg viersus researdh 

It^seems to nrie.that much of the emphasis on the 
primacy of outcomes in quality assessment arises 
from a fundamental cbnfusion between research 
and monitoring. In research, new knowledge is 
sought about the relationship between outcome 
Y^c*^- In assessment and monitoring, exist- 



Nothing I have said so 'far should .be taken to 
mean that outcomes ai^ not important in quality 
assessment and monitdring. ^uite the reverse is 
true: Let me count the ways. 

' 1. Outcomes, usually undesirable ones> can be used 
as a method^ of sampling or screening in order 
to increase the yield of process assessment by 
concentrating on cases with such .^outcomes. 
PeiJ^ps the clearest example of this strategy il 
to he found in the **problem-status outcomeV , 
method described by Mu^hlin et al.® It is also vis- 
?Ale as an important element in the Performance^ 
Evaluatidn Procedure (PEP) advocated by the 
^ Joint Commission on Accreditation of IJospi- 
tals.'o . 

2. Outcon\es can be vtsed as a proxy for elements 
of process wh\ch are difficult* to measure, or 

^ about which information is hard to get or is ab- 
sent, provided the caudal linkage between out- 
come and process is reasonably well established. 

3. Outcome items can also serve as a supplement to 
monitoring process in order to ensure that im- 
portant process elements have not been over- 
loc/ked. In this case, they provide an added tier, 
so to speak, in the si^eillance system. 

4. Outcomes can also serve as a feedback 
^ ./^mechanism that may lead to questions not only 

about whether certafn process 'elements are 

13 ■ ' . 
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inadequately represented in qjonitpring but, more 
fundamentally, aboujt assumed relaUonships b-^- 
tween process and outcome. previously dis--« 

' cussed,' this latter is the point where- monitoring 
and /research are most likely to intersect. 

5. The inclusion of outcome assessnient can also 
serve to reinforce problem-oriented* manage- 
ment, or "management by objectives/'^* * 

6. Attention to the> attainment of prestated out- , 
* comes can motivate a more serious exahiinatio/i 

of process and be a powerful spur^^eform. In 
Williamsons work on **health accounting/' for 
example, one finds not only an-imfilicit prefer- 
ence for management by objectives, bu^, also, a 
. deliberate reliance on cohfrontation with fa|i>re 
iirord^r to jolt physicians into action. r 

7. jOut«omes reflect the impact of all the elemrents 

go into^caK,.and of much else besides. 

^hey iiave, theYefore, an integrative , property 
\tfhich allows them to represent the contributibn 

*ol all the healtfi professionals to patient care. 
They also fhclude the contribution of the patient- 
to his own care. Unfortunately, this useful abil- 
ity to pull together all these influences is also a 
weakness, if one^^wishes to explain how the ob- 
served outcomes come about." 

8. Outcomes can not only include the patient's con- 
tribution to care, but also provide a means to^en- 
list the client in the process of quality assessment 
and monitoring. The specification of the techni- 
cal elements ofe process is an esoteric profes- 
sional enterprise about which the client can only 
have rudimentary and, possibly, distorted 
knowle4ge- By contrast, the patient ,has a great 
deal to say about the interpersonal component 
in the process of care. He has at least as much to 
say about the outcomes he expects, to what ex- 
tent these are attained, and how different out- 
comes are valued relative to-^ach other and to 
Aie costs incurred in attaining them. 

|f I have been successful in my exploration of 
the role of process and outcome in quality assess- 
ment, one should never again hear a preference 
for one, over the other, except by the poorly in- 
formal. But, I am also realistic enough to know 
how forlorn a hope this is. JNo doubt the debate 
will go on. In the meantime, it' may be useful to 
present some general principles that govern the 
use of outcomes in quality assessment. 
" 1. Obviously, the outconjes selected should be 
relevant to the goals of health care in apy par- 
ticular situation. 
- 2. This alio means that tht outcomes must be 
achievable by good care and that the in- 
strumentalities needed are available to, and 
under the control of, those whose performance 
is being assessed. ^ 
3. The outcomes must be. specifiable in mag- 
y nitude, frequency and timing. The paucity of 
information about the cotirse of untreated and 



treated disease may mak? it impossible to 
sptcify accurately future outcomes, limiting 
their use in certain^assessment schenies; ^ 

4, The duration of outcomes as well as their 
" magnitude must be taken into account. Hence 

the emphasis on a long-term perspective, often ^ 
'^as long as a lifetime, inf the measurement of 
health status.^** 

5. As*a corollary to the above, one needs to 
consider %he possibility of tradi-dffs between 
magnitude and duration for any one outcome,; 

jand among severaj outcomes. For examplil a 
• shqrt^r^life at a higher level oF funftipn may 
have; to be weighed against a longer lifc-^bur- 
^ clened with greater disability. ,^ - 

§; As an(^Rcr corollary, accurate information on'^ 
the fvant oU«^mes must be available and 
the^^^uRonfts must 'be subject to feasp.nably 
' precise measurement. Since deTinitive; out- 
• comes rfiay ^ot appear except after long 
periods of tinrie, information may be hard to 
get: • / 

7. It is necessary to examine the consequences not 
only of taking a specified action but^ also, of 
hot, taking such actio*, 'in order to get a 
(^omplete picture. Thus, in evaluating the ef- 
fectiveness of ^ surgicarprocedure, ix^ is neces- 
sary to follow not only those who have had, 
operations but also those who might have been 
candidates but were not. ; ^ ' 

8. The attainment of outcoH^es cannot stand 
alone as a measure of success. The means used 
in achieving these outcomes have also to be 
considered, unless it is assumei4 that resources 
are unlimited, which is far from true. This 
simple truth has^^only occasionally been ap-, 
preciated in assessments of quality, as distinct 
from utilization review;^* but it has received 
much attention in program evaluation, under 
the heading of cost-effectiveness and cost- 
benefit analysis.^ In this context, it is useful to 
remember that' the costs are not only in re- 
sources expended or ri^ks taken; any assault 
on established values and social norms is also a 
cost which deserves serious attention, unless it^ 
iy regarded as desirable and considered to be a 
benefit, 

9. At an equally fundamental level, the problem 
of attribiuion has to be handled. Outcomes, 
whether ^od or bad, must be attributable, 
first, to medical care and, then,Mo the per- 
formlance of those under ^ssessmept. Natu- 
rally, the longer the time that has elapsed be- 
tween a specifijed activity and its consequence, 
the m'ore opportunity there has been for the 
intervention of other factors, and the more dif- 
ficult it becomes to assign responsibility for ob- 
served outcomes. / 

10. To conclude this decalogue, when adverse out- 
comes are used to assess quality, one must al- 

14. ' 



most always examine the antecedent proces^of 
care to. And out Wh?t went wrong, and how it 
might be corrected. The search for cauSts and 
remedies will often lead, evei; beyond process, 
to an examtnadoh of tKe^ji^ural characterise . 
tics that have encouFagedJp^scouraged speci- 
fied behaviors.^ The quality :bf care c^nnqt be 

^ fiilly comprehended 6r successfull)--«44ured 
without understanding how structure infli#: 
ences pi:oc^ss,.and process influences outcome^ \ 
No'^tier where one starts in this chai< one * 

|must ^ihimately deal with it as a \vhole. 
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Proposals for research 
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. 1 begap this paper with a general frame of ref- 
erejice hoping -that it would give meaning and 
perspective to the listing of research proposals; 
but, also, that it wduld generate a correspondmg 
classification. Unfortunately, no truly satisfactory 
classification was found? because there is a great 
dearof overlap among the several categories that 

'emerged. Nevertheless. I hope that the following 
IS a reasonably orderly presentation in which the 
reader can discern the major features described in 
the' lntroduct^n. There is also some attempt to 
have a very rough progression from studies deal- 
ing 'with basic concepts and measurement tools, to 

■ those that deal with implemenlatitjn; but too much 
should not be .made of this. Finally, there Is the 
problem of specification. Obviously, it is not possi- 
ble to be very specific in a general review like this. 
The text only sketches areas for research. But, 
whenever I could, I have cited examples of re- 
ported studies that might serve as more detailed 
models. In these the reader will find not only con- 
crete embodiments of the more general descrip- 
tions in the text, but also, I hope, the raw material 
and inspiration that generates new research. Since 
the studies cited are meant only to be illustrative, 
the list of references does not serve as a systematic 

bibliography. . 

-.■ - 

Basic explorations and studies of what constl- 
tutetf quality 

Speciflcation and assessment by modeling As a 

\ (first step in exploring what cdnstitiites quality, at 
)ieast in the technical sense, diagnostic and 
\Jherapeutic management should be described and 
assessed as planned and sequenced activities or 
strategies. The models of such strategies that^ 
emerge will have a ciertain newness, although they 
' use only existing knowledge and opinion. This in- 
formation is obtained from reviews of the litera- 
ture, from retrosp<lctive review of case records, 
from t|ie opinions and values of expert clinicians, 
/and from reasonable extrapolations from all^f the 
above. The models that are constructed, m the 
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form of algorithms or decision trees,- are not only 
more precise and realis(ic representauons of what 
is considered to be good c^re, but also provide the 
opportunity to- test alternafKve strategies and to . 
confirm or modify normatiV standards. Such 
formal testing is/necessan' bdfc?.use in these com- 
plex situations, the best course of action that intui- 
tion dictates may not be the best indicated bj deci- 
sion analysis. ... 

jlI The usefulness of this effort, then, is-that it pro- 
vides the basis for formulating criteria for assess- 
ing the quality of care. The models are also an im- 
portant tool in medical education, as" a vehyjle for 
specifying and communicating to studer^ the in- 
tellectual operations that constitut^ clin^l judg- 
-ment. Moreover, th/ attempt to construcf such 
models reveals defipfencies in current information, 

• ideiitifies the-m^t critical defects, and suggests 
the research needed to obtain the required 

information. . --r» 

Illustrative- examples of this approach to the. 
exploration of diagnostic strategies may be found 
' in the work of McNeil et al. on scteening 
hypertension"* and on diagnosing pulmonary efrij 
holism; >^ and in the work of Neutra on the 
decision to operate for apgendicids-.'s Examples of 
the specifttation and testing of more complete 
strategies that include treatment as well as diag- 
nosis are provided by Schwartz et ak for essential 
hypertepsieiahd renal artery disease;'' by McNeil 
and Adelstehi for hypertensive renovascular 
disease;" and by Tompkins et al. for acute 

pharyngitis.^ . 

A review of these publications illustrates the im- 
portance of costs as a factor ii* the analysis. Non- 
monetary cost in the form of risk is always a factor. 
In many instances monetary cost also figures 
prominently^ for example in the papers by McNeil 
et al." " and Tompkins et al.'' The problem of 
placing values on' the consequences of various 
courses of action bedevils the analysis, by Schwartz 
et al.'» Methods for valuing the quality of life, m 
addition to its mere duration, are briefly discussed 

by Abt." " , / 

Even a nodding acquaintance with this wprk 
suggests opportunities for further research, be- 



il3 extension to cover a wider range of 
Ij^^ical situations. ^One line of inquiry is to go 
jj^^tid detf*''"i'^ing how to best arrive at a certain 



V'^'Tfi^li^^*^ studying jj^e^rnanner ip which cases 

* 3^??^rcscnt c!inica% identifiable problems can be 
j^'^Rlied several diagnoses. Jhe work of Ginsberg 
stu^ serve an illustration." Another interesting 

woij'd'be comparing th'e stepwise, logically 
r^^iinonious approacfi to diagnosis by tho "shot- 

* nu*^*' approach, which uses a very much larger 
, *^bcr of at least as a first step, to quickly 
y ^^^w d0^"^ the realistic ahernatives. The possi- 
sh^ .""^^ and in the discovery of illness 
j^^^ld be weighed against the monetary and non- 

> ^^etary costs. \ v_ • 

^^Wrical testing of strategies of^cari Modeling, 
'^tter how ingenious or ci'eative, is limited by 
' jl^^^ currently known or believed about the rela- 
i.^'^^hip between process and outcome. Additional 



Pledge niust come frOm empirical studies 



pjj^^^gies of management. This may bcf^ccom- 
exD^^^ by observing what might be called ^'natural 
^Q*^^iment5"; but defininve conclusions can only 
Ks^fA ^^^^ controlled frials, which cap bg re- 
meri-^^ as an extension of research in clinical 
^ ^*^ine. But clinical research is primarily cc^n- *' 

f^^d with the comparative effectiveness of alter- 
and^^^' relative costliness, or the balance of costs 
benefits, is seldom an issUe. When clirtical 
^^^nd ^^P^^^ ^^^^^ scope to include not only risks 
ben ^"^^f td health, but also monetary costs and 
fj^^^f>t&, tli^y address questions of social Rolicy 
ass ^'^^ essential ingredients in quality 

^^ment. ^ ' 

-J algorithms that currently define acceptable 
lon^^*^^ derive their authority from expert of^- 
The vtork of Meyers et al. is ^ne rather 
example that is a reasonably direct em- 
dia^^' test of such an algprithm: one fof the 
^^^/^Osis of meningitis in children. ^3 Som^ of the 
3l ^ cited earlier, for example that of McNeil et ^ 
^ the diagnosis* of pulmonary embolism, can^ 
j.^j,^^%arded the same light, since it, uses patient 
j^ r^s, though retroactively, to construct a diag- 
Sturn ^^""a^^^y**^ ^ mo(Jest prospective study by 
pl^ r^Vant ^nd Stern that tests the ability of 
chor^"^"^ to predict the finding of a stone on 
^^^j ^Cystography -is relevant becaose-analogous 
1^^^. jf^es arc "sed in de^sion analysis, when no 
j^Q^^V data are available.^^ An exam'ple of a much 
^^^\nibitious clinical trial that te^ts conse- 
foun addi^onal to medical outcomes may be 
/ econ thfsJJ^ork of Fiachaud and Weddeil on the 
gjy^ ^^ics' aJ' treating varicose vein^.^* In this 
^\^^ the^relative merits of injection-conrtpression 

.^^herapy ^"^^ surgery were compared inpa 

i^n experiment, using as criteria the condi- 



elements of care or the more cdmple 
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ex^ 




tioti of the limb at the end of three years, mortality 
from the procedure, the occurrence of immediate 
conrplications, the loss of patients due to non- 
attendance, the current monetary costs of the 
procedures, and their indirect co^^ as represented 
by travel time for trieatments, days/ taken off workfe^ 
and loss o^f earnings. There was no attempt to 
construct a singl^^ieasure for costs^and another 
for benefits, but the advantage was clearly with 
sclerotherapy. The implications of such studies to 
N^uahty assessment and to social policy are obvious 
and important, Assuming that their findipgs can be 
accept^ as valid. ^ 



Increments of cost^and of quality As I tried to 
show inrthe introductory section of this paper, the 
relationship between in^ements of^quality and of 
cost is a matter that deseXes special attention. One 
way of studying it is through modeling and simula- 
tion. Using this method, McNeil et al. have esti- 
mated that it cosCs $8,485 to identify the last one 
out of 20 case^ of pulmonary embolism in a'young 
adult population with pleuritic pain, wF\ereas the 
preceding 19 cases could be identified at an aver- 
age cost of $595 per case.^^ Even more striking are 
the cosrestimates of Neuhauser and Lewicki fcfr a« 
procgiku-e for finding cases of cancer of the colon 
ln-€hypothetical population of 10,000 persons 40 
years old or older.^* The procedure is to first do a' 
stool giiaiac test and afterwards to x-ray those who 
show at least one positive test. What is vafried is/he 
number of successive guaiac tests thfit are incluUed 
in the first step. Obvimisly, as the number of luch 
tests is increased, the number of cases who show^at 
least one positive test increases; more cases wit! 
cancer are detecte^, but there are also many more 
false alarms to be allayed by subsequent barium 
enema. Accordingly, the cost of finding an addi- 
tional case of cancer rises steeply as the "quality" 
of care, judged by the ntimber of stool guaiacs 
j pr^rscribed, increases; so much^so that the cost of 
finding the, additional^ case^is. esdmated at an as- 
tounding $47 million when 6 tests are done, 
whereas it is only about $1,000 when one test is 
used as the screen, and about $49,000 when three' 
tests are used. The values that are generated by 
these models are, of course, heavily influenced by 
tb^ models* assumptions; and they could be inac- 
*curaj,e. My intent is to emphasize not the findings, 
but the nature of the method and its utility. 

As an extension of such explorations, under es- 
sentially hypothetical conditions, itynight be useful 
■ to have clinicians formulate a set oj^ .increasingly 
stringent criteria of the quality of management, as 
well as an estimate of the gains in health expected 
from such stepwise increments of quality. Cost es- 
timates could then be made, and the expected 
behefit compared to the cost. Hardwjck et al. im- 
'^pleniented what is, in effect, the first phase of such 
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a procedure 'when they asked house staffand gen- 
eral praclitioners to specify the diagnosite tests 
they would perform for specified conditions (1) ^ 
an absolute minijnum, (2) routinely. (3) if llie case 
werc^to be presented at a medical grand lotinds. 
and (4) if it were to be included ^SoTO research 
study.*^ . > , 

Npedless to say, empirical studies are necessary 
to establish the relationships between quality and 
cost. A study of secular trends in the inputs into 
care for carefully speciTred conditiojis as compared • 
to the outcomes of suth care could be qne initial 

^approacfi. A gooa example is the study by Martin 
et al. of inputs into coronary care compared to the 
mortality during hospital stay." While such studies 
are of greal interest, contemporaneous compari- 
sons of costs for similar services at differenj levels 
of quality may be 'more credible. In one such 
study, Jackson and ^mith found no relationship 
between estimates of the quality of pharmaceutical 

• services in community pharmacies and the charge 
per pres^riptK>A.?,» Similarly, Schroeder et al. 
foijud noS;l«5r relationship between the cost of 
laboratory services ordered by residents and the 
assessment of the competence of those «sidents by 
^heir supervisors, even though there >rere wide 
variations in both estimate^.?" Rubenstein et al. 
have reported findings that suggest diminishing 
returns in outcome ratings with increments in the 
quality of the process, of '^are.a* These are only a 
handfiil of studies, but they show the way for 



more. 
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Studies of the client-practitioner relationship 

The research I *^have . proposed so^ far defines 
quality in terms of the technical elements in clini- 
cal judgment. There is a corresponding universe 
of concern with the judicious management of the 
interpersonal relationship, between clients and 
• practitioners. Here, also, there are differing 
modes of interaction aiid varying styles and 
strategies of management that are more or less 
successful in achieving desirable outcomes. These 
outcomes include satisfaction, knowledge, change 
in attitudes and behavjor, including compliance 
with recommendations. All these may< in time, be 
related to success or failure in altering health 
status. Fundamentally, there is little difference 
between studying the effects of two drugs on 
hypertehsion and a study of the effects of two ways 
df managing the interpersonal relationship on 
compliance with the drug regimen itself. Both are 
clinical trials, though they may draw on different 
bodies of theory and concepts. But, despite the 
similarity, the study of the effect of drugs is 
considered squarely within the domain of clinical 
practice, whereas no one is quite sure where the 
study of the interpersonal relationship belongs. 
Ideally, clinicians should be as interested in one as 



in the other, and actively engaged in research into 

both." . " . ■ u 1- \* 

The literature pertaining to the client\ 
jiractitioner relationship is too vast tp permit a 
quick summary .-A text book review by Bloom and 
Wilson serves as a useful introducti:on." Lebow 
has reviewed the literature on patient satisfaction, , 
discussed some of the problems of method, and 
made suggestions for further research." A specific 
example of a clinical trial that examines the conse- 
quences of a change in the cfient-prpcutioner in- 
teraction is the study by Inui et al.,:which shows 
cbnsiderable improvement.in the control of hyper- 
•tension wh.enV>hysicians shift their attention from.' 
the tpanifestations of the disease to th%behavior of 
the patients." These encouraging findings can be 
placed against the discouraging experience in 
another, dftnical trial of the lack of success with 
either of two strategies, "augmented Convenience" 
or "mastery learning," to improve medication 
compliance in hypertension.^' 



Views of wha» constitutes quality The research we 
have reviewed and ppposed so far is expected tp 
give a more/precise answer to' the question: What 
constitutes quality? Another approach' is to study 
the opinions held,' on this subject by clients, ad- 
ministrators and professionals. A* comparison 
should be made of the ittrijjutes that are believed 
to constitutfe-^uality and of their rankings in 
importance. The findings would be germane not 
only to quality assessment, but also to under- 
standing problems in system performance. ' 

An example that fits precisely in this category is 
the study by Smith and Metzner of the opinions of 
patients, physicians and nurses on what constitutes 
quality of care in prepaid group practice." Mar- 
ram's work on how much credence nurses are will- 
ing to give to the opinions of patients concerning 
different aspects of nursing performance suggests 
an interesting topic for further study." The litera- 
ture on patient satisfaction to which we have al- 
ready referred includes much material from which 
inferences may be drawn about howxclients view 
the quality of.care. In this respect, patient satisfac- 
tion has an interesting dual nature: it can be re- 
garded not only as an outcome of care, but also as 
a judgment by the client on the quality of care. 



Continuity and coordination as attributes of 
quality There is a general presumption that the 
continuity afid coordinatioiTbf care are impottant 
considerations in the assesAment of quality, even 
though it is not quite clear what these phenomena 
are, or how they fit into a formulation of what 
quality is. De Geyndt accords them a central place 
in the definition of quality by consideri/lg th^m to 
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be aspects of the 'if^rocess" of A^as Jisfinct from 
itsJ*content/''* I have preferrerffe regard^them as 
an aspect of the organization of what I h^ve called 
^process.**'® More^important than these formal 
/distincfkk^s is the pricise deflnition and measure- 
ment of these^atCrih/utes, and the study of th^ir 
contribution to the outcomes of care. Empirical 
^dy should also include the degree^ to which 
existing differences in, or purposive manipulations o 
of, certafn structural features odng' about differ- 
/ences in continjiiity and coordin^ion of care. I 
would also like to see/ a test of the hypothesis that 
planned transfers of the responsibility o£ care 
from one physician t© another ' might actually 
improve care by crating an opportunity to assess 
it anew, and by making th^ performance of physi- 
cians more visible among colleagues. ^ 

Important contributions to a- more pi^ecise defi- 
nition of the concept of continuity may be found 
in papers by Shorten*" and by Bass and Windle.*' 
How^ever, the tendency to o]»erai|i'onalize con- 
tinuity to mean care by one physician or a single 
source of care, except by referral, does not seem to 
me to capture the essence of the concept of 
coritinutty. Almost all the studies I have seen have 
used this definition. Bass and Vyindle tried to 
assess the relatedness of past to present care.*' 
Shorr and Nutting defined continuity as the com- 
pletion of a needed sequence of /are.** Becker et 
al. describe a well-controlled tria/bf the effects df~r 
having children see the same physician at each 
clinic visit, and give a gobd review ot i\ie relevant 
literature.*^ 



Description 6l prevalent patterns and strategies 
of care 

In a (Previous section we discussed th^modeling 
and testing of strategies of care. In this section we 
focus on a complement to the research described 
earlier, namely on the identification of how physi- 
cians behave in the real world, described in terms 
of the elements of care, as well as bundles and 
configuration^ of such elements. Here we seek to 
describe what goes on, what factors , influence it, 
and what the consequences seem to be. 



Studies of the elements of clinical behavior Much . 
mOTq information is needed about differences in 
the patterns of care among practiti6nei:s ,wit/iin a 
given setting and across settings. >Ve need to 
understand wh^t factors are re^onsible for such 
differences and what the costs and other conse-. 
quenc^s are. The factors invplved could incmde 
personality attributes, knowledge, training and, 
socialization, position within an organization, re- 
. sponse to role models within the organization, and 
v.^ ial incentives. ' 

ERIC' 



Several studies of the use of laboratory and . 
other diagnostic procedures cin be cited as a good 
example of this type of research. The findings of 
Childs and Hunter suggest that monetary return 
on JfcMnvestment in an x-ray machine may be a 
factor in recommending radiological procedures.*® 
On the contrary, the persistence of large differ- 
ences in the use of laboratory and. other services in 
^tUi^s where there is no direct financi^incentive 
. to use such services,** or where the incentive is the 
same for all ph/sicians, suggests that other fac- 
ibrs are equaUy or more important. Schroeder and 
his associates in a series of studies have looked into 
the correlate^bf 'differences in. laboratory use and 
showji that these differences ^re net related to dif- 
' fcrences in compc^tcnce,^** or to differences in out- 
comes, or "productivity";** and. that^physicians 
may respond to a **cost audit" by reducing labora- 
tory use." 

IdeWtifli^ion of . styles and strategies I liave 
already defined a strategy as a plan for action, and 
oi^cussed the i^jportance of cTealing with ^rStegies 
in the definition and assessment of quality. A 
*'§tyle" nfjay be defined as a habitual preference for 
certain .modes of decision making which would 
manifest itself in components of strategies or 
strategies as a whole. For example, a physician may 
exhibit- a persistent and pervasive preference for 
errors of commission over errors of 'omission, pne 
manifestation of which may be a large redundancy 
in gathering information"^ He may give evidence of 
more than usually routinized or stereotyped be- 
havior. These and other persistent yet undefined 
/ propensities require precise conceptual formula- 
>aion ^nd empirical study. ^. 
y The notions of style and strategy have applica- / 
"tion beyond the solution of clinical probleilft. I 
have already suggested that they can also be used 
to study the client-practitioner relationship. They 
also apply to ihe way in which a' Drjjbtitioner 
manages an entiW case load, hoping to^athieve the 
most efficient allocation of his time, attention,^ and 
other resources among competing calls on them. 

Styles and strategies can be inferred from physi- 
cian behavior in real-life situations or under more 
artificial test conditions. Information on the 
rationale employed u>y the physician can be ob-' 
tained more directlylby having him explain, as he 
works, the reasons for doing what he does. Fattu 
has summarized someVf the early work using this 
method, known as refJei^nn p^lee, as wc;ll as other 
methods in studying prc>Wem solving.*® A more 
recent exaVnple of the use of reflexion parlee in 
exploring clinical decision making has been re- 
ported by Kleinmuntz.*^ A study of the diagnostic 
process by Leaper fet a), notes variations in the 
degree to which the^chnical interview is either 
"stereotyped" or *^apted" to the problems of 
J ^ch patient, as inferred by an observer.*® ^ 
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Obviously, strategics that are identified through 
empirical study become Candidates for testing, as 
described in the opening section on research pro- 
posals, I silscvb«lieve that strategies used by "good" 
pjkysicians are a *nore valid basis for the formula- 
tion of e?c^li?tt process criteria than is the practice 
of having physicians list all the things that sbould 
be done for cases with a specified diagfiosis. 

" ' ^" 

Compaiiaons- of ' norms with practice The litera- ^ 
ture teems with observations that physicians fall 
short of the normative standards of care. In some 
cases'it has, bee» suggested that physician^o not' 
follows the staW&ards which they tHemselve^, as a 
group, have Q&rmulated.** Some ha\^ claimed that 
mahy errors in care are due to inattention by the ^ 
overworked physician. Others have shown that 
part of the deficiency in performance is jjue to 
lack of knowledge, while another part is du,e to not" 
acting on what is known. In my opinion, the 
explicit criteria lists which are often used to judge 
performance are. themselves, pften faulty'. One 
important inadequacy is that they fail to take Ac- 
count of the many contingencies, including multi- 
ple diagnoses, that modify the strategies i5f man- 
agement. Moreover, it may be inappropriate to 
-apply to office practice, especially to that of the 
generalist, criteria derived from strategies gf man- 
agertient that are suitable in acadefnic settings 
wllere highly specialized physician^ are involved in 
the^Iution of difficult diagnostic and therapeutic 
problems. 

Unfortunately, much of thd above is ortly conjec- 
ture; aiid there is an urgent need for studies tha^t 
attempt to understand why physicians conduct 
care in the way they do, before passing judgment 
that what they do is inappropriate. 

. . ■ ■ ■ .,r : 

homogeneity and heterogeijfety of performance 

An interesting question witl/many practical inipli- 
cations is whether physicians and other practition- 
ers perform equally well across a range of activities 
ind functions, or whether they do* well in some 
and not so well in others. This involves examining 
the h^ogeneity of performance in the practice of 
indivicKial physicians, as well as the ability of an 
institution to sxeduce variation in practice across 
physicians. 

The relevance of this issue to quality assessment 
is most apparent when it comes to sampling. In 
one method, the "tracer methodology" proposed 
by Ressner, there is an explicit assumptian that the' 
performance of an entire system* can be mapped 
by judicious selection of a small number of condi- 
tions that can stand for all the rest." In many 
other studies, when a ^Hial 1 number of diseases or 
conditions are selected fobsassessment. there is a 
similar assumption, even though it may not be 



explicit, or occupy such a central place in the 
design of the method. <y 4 

Kessner has tested his assumption Of 
horaogeneity aqji found it, at befet. frail." L)(pns 
and Payne have reviewed the literature and done 
further .testing of the degree of compliance with 
'normative standards of management by individual 
physicians, acrpss diagnoses, in office practice,*^ 
and in hospiul practice." Intercorrelations tended 
to be low. There was. however, a sugg|fetioi> diat^ 
greater homogeneity might be found in, the work" 
o/ physician subgroups Vho have^a more restricted 
domain. This clustering ^as also fouhd in a Jludy 
of th,e office care of a set of preventive and illness 
situations in children. Homogeneity of perform- ^ 
ance was reasonably high within each of these 1 
categories of conditions, and it w^ higher foj 
pediatricians than for family physicians.** j ^ 



The epidemiology of quality 

Variatiorts in the quality of care are nilt simply a 
random phenomenon. They are highly p^terned, 
and responsive to caHsative factors tt^at we. need to 
identify, and understand if the quality of care is to 
be^successfuUy safeguarded. The first step in this 
exploration is to answer the classic questions of 
P^demiological investigation: How much? Who? 
\«iere? and When? The results of thes^ observa- 
tions mw suggest answers to the most cridcal of all 
q^uestioljs: Why?:The causal hypotl\eses that 
emerge could, then, be tested through more^lgor- 
ous objservational studies and confirmed by actual 
experimejitatidn. But, fpr now. even^the simplest - 
of descriptive studies would add a great deal to the 
lithe we know. _ > 

On a larger scale, we can r/bvi say ^ost nothings 
about the quality of care for the natfon as a whole, 
or for reasonably large populations in th^ir natu- 
ral^bitats^.oVher than what can be inferred from 
; cHide mortality, morbidity and utilization data. 
The one exception to this generalization that I 
know about ts the study of a segment of car^ for 
the residents of Hawaii by Payne and his as- 
sociates.'^ ^e are similarly in the dark about time 
trends: Is the quality of care improving, and how 
rapidly? This question is difficult to answer be- 
cau?t it requires the separation of two phe- 
nomena: changes in the science and, technology o£ 
medicine, and changes in the a|3plication of that 
science and technology. Both phenomena need to 
be assessed. 

The epidemiglogy of quality can be viewed as 
manifesting itself in two populations: (1 the pro- 
,viders. and (2) the clients. Obviously, these are not 
two sep^rate compartments. Variations in the qual- 
ity of care received by clients are probably largely 
due to the kirfds of providers who care for them. 1 
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to what extent the reverse could also hr 



Tne« epidemiology of qualit]^ among providers 

^ The studies that might belong in this category 
overlap^ith those I have already described in the 
. ptfvious section on Description of Prevalent Pat- 
. ferns. If there'is a'^istinction at all, in the earlier 
section 1 ^included%iudies that de^lt with the 
detailed Jton tent ofv c^re and , the rationale that 
^ explained differences in that content4Rere, we are 
more interesti^d in who provides care of godd, bad 
or indiffer^t quality, 4rtd under what circum- 
stances. Much of the literature of quality assess- 
me/it deals with this queslipn; but the restricted 
scope of most studies, and limitations in their 
design and analytic methods, makes generalization 
hazardous. It is clear, hoyrever,' that pjcrformance 
is related to attributes of practitioners, attributes 
df the organiz^ional settings in Which they work, 
and interictibn^ between the two. Among the 
attributes of the providers are education, training, 
specializadon and length of pracuce. The role of 
persdnality characteristics including motivation, 
while suspected to be large, has not recdiJed much 
attention, except in studies of the academic per- 
formancle of medical studer>is. Among the attrib- 
utes of organizational settings, usually hospitals, 
that have been found to.be influential, perhaps the 
• most important has been affiliadon with a medical, 
^ school. Other attributes are involvement in resi- 
, dency and internship training, auspices, staffing, 
financing, size, and organizational contrRs on 
suff appointments -and activities. Unfortunately,- 
diere is far frqipi unanimity on whether these 
factors are influential and what their effects are. 
For example, we are still not certain whether the 
organization of physicians into prepaid groups 
^results in better quality, and, if so, to what extent. 
^The number of studies that have attempted ac- 
tually to quantify the influence of each factor 
separately an« in combination jivith others is par- 
ticularly small. Whenever this has been done, what 
is/most impressive is the very small amount of 
varj^ce explained by the variables in the analysis, 
suggesting that we still know very littie about the 
determinants of the quaKty of care. 
, The literature relevant to this section is so large 
that even a partial review would be a herculean 
task which I shall not attempt. Only some exam- 
ples will be given. Among the earlier studies 
particularly outstaying is the work of Peterson et 
al."» and of Morehead et al.«« Among the more 
recent work is the Hawaii study by Payne and his 
ass^ciates,*^ and the study of Medicaid benefici- 
aries in New Mexico by Brooke and Williams. Of 
particular interest in the latter is the special atten- 
troh to "outlier*' physicians, those whose perform- 
O as markedly deviant. Further studies of 
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the characteristics of physicians that perform par-^ 
ticularly well or badly could be a useful way of 
generating hypotheses about the determinants of 
performance. J ^ * 

^ Among the recent studies that have attempted to 
j mMsure the magnitude of the effect of each of 
f several variables on iperfx^ijnance arevthose bf 
Rhee^* and of the Stanford University group 
responsible for the Institutional^ifferences Study. 
" Rhee used datia from the study by Payne et al.*^ 
in which the dependent variable was a perform- 
ance score based on compliance with explicit profc^^ 
ess criteria. Notable among his findings was* the 
large effect of hospital^cliaracteristics compared to' 
the effect of specializa(tion artd of organization into 
larg^,"^ultispecialty groups; and, eveitVpiore 
striking, is the magnitude' of the iinexplained 
variance.' In their study on post-operative surgical 
mortality and morbidity the StanfoVd group, not* 
only found unexplained variance of a similar 
order of magnitude, but also failed tcr confirm the 
effect of factors usually thought to be conducive to 
quaUty, for example hospital size and university 
affiliation. Both of these studies can serve as 
models for future work. The Ihstitutional Differ- 
ences Study is particularly notable for the methods 
it developed to control for risk factors that influ- 
ence the outcome of sWgery and td specify and 
^measure organizational^ variables. Its findings 
Vrould perhaps be better uYiderstood .if samples of 
records in the hospitals involved were to be sub- 
jected to an assessment of the profess df care. I 
would accord su,ch a study a high priority in this 
prospectus. * I 

The correlations between organizational charac- 
teristics and performance are, qf course, extremely 
important for system design. It would Jbe interest- 
ing, therefore, to go- one step further and deter- 
mine whether the same physicians wljl* alter thei 
behavior when placed in differeftl settiiSgs. 



Epidemiology of quality among clients Unlil^ the 
emphasis on provider characteristics, there has 
been very little attention to client characteristrcs in 
studies of the quality of care. Although the receipt 
of quality care by disadvantaged populations 'i^ a 
matter of great social concern, and there is mvich 
public debate about it, the information bearing on 
the question is indirect. It deals mainly with 
differences in levels and patterns of utilization, in 
sources of care, and in morbidity and mortality 
data. Without minimizing the relevance of such 
information, it would be useful to have more 
direct and definitive assessments of the quality of 
care received by persons differentiated by age, sex, 
educations, color, income, occupation and other 
demographic and socioeconomic variables. Much 
of the differences would probably be related to 
differences in sources of care. But a question that 
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\ccd8 to receive special attention is whether th^ 
same institution, and the same practitioner, ^Ve 
different types ort^rc^o patients with similaf 
medkral conditions because of differences in 
demographic arfd socioeconomic characteristics of 
the latter. Some adaptations of care to such 
characiei'istics aref of course, not onjyJI^Uimate 
but, also, desirable and necessary. The issue/is to 
determine \^ietber the adaptations are piade to 
maintain a high level of quality or whether quality 
suffers. 

The literature having a bearing biT^uality of 
care for disadvantage;^ populations-K^s^ been re- 
viewed by Broo)fr-aTid Williams/^^In anotlier publi- 
cation they d^ribe their^wn fitnlings in a study 
of Medicaid l^^eficiaries.^" Lyons and Payne h— 
described the ri^lationship ^etween age and 
quality of care^'in their studies irt Hawaii 
Kessner et al.> have described the Vetationship 
between system performance and various demo- 
graphic aiUl socioeconomic factors, as well as 
source of care, revealed by an application |of the 
•tracer" method in selected populations m the 
Washington, D.C. area,®' Griner and Liptzin have 
described the effects of patient characteristics such 
as age, insurance stetus and ward or private 
accomipodation on the use of laboratory tests in a 
teaching hospital.^* ^ , , 

The identrfrcation of time trends is an important 
tool in epidejfiiolpgical analysis that has been 
seldom employed in studies of quality. Hence, two 
studies that have information on this subject are 
particularly interesting. The first is a study of 
survival after cancer pP the cervix uteri treiated a 
decadcf apart. The findings suggest that improve- 
ments have been due mainly to a diffusion of 
knowledge frqm niiybr centers to commiinity hos- 
jJltals, and to a much lesser extent 'io an in^prove- 
ment in medical science." The second is a study of 
maternal mortality in Michigan from 1950 to 1971 
showing that, in spite of specjacular declines in 
nftirtalfty, the percent of deaths\onsidered "pre- 
• veaiable" by the Staters maternal mortality com- 
^^mittee Kas increased markedly, from about 60 
percent to about 80 percent.^* A retroactive reas- 
sessment of the Committee s file of cases, applying 
current standards of preventability could be very 
revealing when compared to the contemporaneous 
' assessments* 

Most of the work on differences in the manage- 
ment of panents by the same provider, whether an 
institution or an individiial practitioner, has been 
done in the field of psychiatric care. Perhaps the 
best known of these studies is the work of WoV 
lingshead and Redlich in New Haven but there 
are many others. In a later work, Duff and Hoi- 
lingshead showed that such differences in care can 
also be observed in a teaching hospital engaged in 
providing genepal medical care.«« The under- 
sundable reluctance to look into this matter must 



be o4>rcame. No program that provides cafe to 
cliettt* o^ widely varying background* can afford 
to ignor/ the possibility qf discriminatory tiehavior 
in the application 'of caceNnTcobtraveniion of the 
most sacred traditions o^the h|caling professions. 




yTw relationship of structura to proceM or 

the\eader will recognize that the epidemiologi 
' ^al^tu^es skeicheyl out above often deal wlthiob- 
served relationships bcMfreen structural bharao^ 
teristics (attributfes of practitioners and insti^- 
tions) and process, or outcome, or both. The more 
definitive verification of such relationships will r^- ' 
quire controlled experimentation. The major pur- 
pose of such studies is to safeguard and improve 
the quality of care. However, at the* same time, 
they can elucidate the op^rationjpl meaning of cer- 
tain concepts, for example "continuity." To-the ex- 
tent that attributes of slfructure are found to be 
regularly related to perforniance, the more grti- 
ecal use of sUch attributes as measures of quality 
will be moi|c firmljLcaj^lished. 

Development df basic tools for assessment ^ 

Many of the studies mentioned in previous scc-^ 
tions, as well as many still to be described, cannot 
be done well unless certafn basic tools- ai^ avail- 
able. Thus, the refinement of existing instruments, 
and the development and testing of new ones, is a 
necessary part of research in quality assessment. 
At issue are the reliability, validity and cost of the 
basic instruments of assessment. A few of these, 
that 1 consider most important, ^11 be selected for 
further attention. * ^ 



Specincation and measurement of outcomes. I 

have already indicated the ways in' which the 
measurement of outcomes fits into quality assess- 
ment* and monitoring. In this section, 1 shairi 
describe briefly some ways of measuring outcon^gs/} 

One approach is to develop and iise indicators! 
health and social well-being which permit a gen- 
eral oversight of a community of population. This 
approafch is typified by the "sentinel events" pro- 
posed by Rutsun et al.«* A useful area of research 
woulcl be to specify appropriate indicators, de- 
velop methods for data collection, actually imple- 
ment data collection and interpretation, and de- 
termine the usefulness of the information in 
bringing about change. Such a system would, of 
course, have to be adapted to the special needs of 
its users: whether a planning body, a program, or 
an organization that provides care. Any of t^ese 
agencies may need to supplement information 
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which it * coi]^ widi * information from otlitr 
; sources, including census clau and informaiion 

* from. the Na,lional Health Survey. Naturally, the 
^ indiqitor conditions need not only outcomes; a 

* variety of process elemHnt?*can^ also be included. 

* Moreover, the system will be relevant not only to 
determination *of quality, but also ST need and 

^ unm^t need, resource use, and so oru--Many or- 
ganizadons are,^^o doubt, alr^dy invdlved in dau 
gathering activities of this general kind. Perhaps 
the flrst step wou)^ be to review all that is being 

* done, t^ document Its current use, and to assess its 
potential usefulness. It is^mpbrtqnt to rfemember 
that a system overladerr' with data that are not 

.usefuL or are not us^, can be as bad as one that* 
tod little mformation- In any event, a pause for 
reaa^jieisment could be most helpful. 
A second line to pursue is the development of* 

* 'Integrative" Vneasures of health status that can 
represent the outcome cff all the factors that 
influence health. The distinctive features of such a 
measure are that: (ILihe impact qf mortality and 
mqrbidity are combirtcd; (2) mor^kluy is repre- • 
sented by a gradation of mutually exclusive 
categories of dysfunction; (3) dysfunction incliides 
social and psychological, as well as physical* disabil- 
ity; and (4) the several dysfunctional states are 
weighted and summed into a single measure.-Thc 
object is to arrive at a summary representation of 
the quantity, and quality of life of a cohort of indi- 
viduals over a period of time, often a complete life 
span. * 

Elsewhere, I have briefly reviewed the earlier 
stages in the development of this approach 
dirough the work of Sanders. Chiang, Sullivan>>> 
and Fanshell and Bush." Since then, this area of 
endeavor hA^experienced an almost explosive 
growth which includes further work by Bush and 
his associates;*' *^ the work of Torrance,** and the 
work of Gilson and her associates on the Sickness 
Impai:t Profile.** *^ Three collections of papers will 
provide the reader with a concentrated and rea- 
sonably current overview of the field. **'** **^ • 

Perhaps the central problem in the construction 
of integrative measuVes of health status is that of 
valuation. There is need for empirical studies of 
the valuations placed by clients on different de- 
grees and kinds of dysfunction. Expecially in- 
teresting would differences in relative valuation 
by persons who are currently experiencing a par- 
ticular level of disability. The effect of length of 
time in any level of dysfunction should also be 
examined, as suggested by Torrance.** Another 
line of development might be to try out a totally 
different method of valuation, comparison with a 
sUndard population, as proposed by Breslow and 
his associates.*' 

A third line of development is the construction 
of integrative measures, of heahh sutus that are 
— ^•^-on-specific. The global indices described 
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above will probably be found to Ije lacking in sen- * 
, sitiyity and specificity when used ai measures of 
the*quality of ^care. There is an opportunity to 
reAedy these dk^ects by developing analogous 
measures of the ouratidtvMdauality of life of per- 
sons suffering f^»m specific c^iulitiOns, fbr exam- 
^ple a particular cancer. The meiimres of function 
and dysfunction included can theb be Uilored to 
the condition, with^itenfion giv^n to including 
manifestations that can be jwvemed or remedied 
by proper care. The testing of such measures will 
alsQ^provide an opportunity to study the courrse o£^ 

^JHffcss and identify^dditional outcomes:^ihat cstfi 

I serve as measures of quality. 

V A fourth line that miglu4>e pursued under the ' 
^general heading of outcome^ measiires is the de- 
velopment of condidon^specific^indicators of out- 
come. Here, only key indicators are used singly or 
in a profile;' there is no attempt to integrate them, \ 
together with losses from mortality, into a single 
measure, as envisaged in the preceding section. 
But, obviously, there is a relationship between the 
two approaches, since the identification of indi- 
vidual indicatQTMnust be a step iii the construction 
of an integrativ^ measure. \ 

The time ehipsed since^ the institution of care is 
an important classifying variable in outcome 
studies. Accordingly, the indicators of oiXrome 
can be contemporaneous with care, Dr can foUow 
care, in which case they are proximate (short-term) 
or remote (long-term). Short-term outcomes are of 
special usefulness in monitoring becai^ they can 
be used toidentify cases that require further 
study. THs^"* 3 method of venerable ancestry, 
going back 'to the, classic work of Codman,** and 
earlier. Its more recent manifesudons in^the work 
of Williamson and his students have already been 
noted."'* A particularly useful model of the kind 
of research and development needed for con- 
strucdng short-tenm indicators of outcome, is the 0 
work recently reported by Brook et al.^* What still 
remains as an essentially unsolved challenge is the 
development of concurrent monitoring, u^ing out- 
comes during the pr9cess of care. Of course, the ^ 
conduct of care from day to diy is constantly ' 
guided by such outcomes. The difficulty has been 
in adapting this everyday occurrence to a formal 
system of monitorfng, other than direct supervi- 
sion by peers or superiors. SoAie of the recent ad- 
vances irt computer-aided management do^ how- 
ever, sugtest a possible solution.*® What seems, to 
be necessary is the consUnt feeding of selected in- 
formadoh into a computerized system which raises 
an alarm when certain, prespecified, configura- 
tions of events occur or fail to occur during speci-i 
fied time intervals. 

A final line of inquiry derives from adopting a 
definition of quality that includes phenomena such 
as satisfaction, attitudes, opinions, knowledge^ ill- 
ness behavior, and the like. This opens up the 
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whole ava of methods in behavioral research 
which caAbe assessed and implen^ented by inves- 
tigators having the necessary pYeparation. A 
model for such research that is closer to home, may 
be found in the work of Hulkai^nd her awociates, 
first, inyUve development^ of a scale of client satis- 
faction and.^hen. in using that instrtiment to 
study its epidemiology. 

Improvement^ in the medical reeifa^The medi- 
cal record is almost always fife key document 
jwhich contains dhie information for the assessment 
of care. Judgn^nts bf quality are )ieavily influ- 
enced by the natOre of the record. There is also 
the possibility that the record is. itself, influenced 
by quality assessment "activities. •• •^ Unfortunately, 
in spite of its key role in patient care an\l its evalu- 
ation, the record is often inadequate or poorly 



allbwed the traditional content of the niedical rec- 
ordTto dVcUti what is included or not mcjuded in 
assessments of quality. In this way. the recofd con- 
strains the definition of quality, allowing the^tail to 
wag the dog. It seems to me that B is mote reasonj 
able to begin *y defining quality fndependcptly of 
the record and. then, to desigji the record so that, 
alone or in combination with^bther ^pecified 
sources, it can provide the infc^-fnation tOat <;prI*e-> 
sponds to the initial definition of quality. In such 
an enterprise there would be a great. tempUtion to 
demand an im|iossibl6 degree of completeness. 
One must be adamant In resisting it. The more 
reasonable and xrhallenging objective would be to 
define and implement t^e near-minimal set that 
would permit proper management and assessment. 

I have already referred to the uses of com- 
puterized records in computer-aided management, 
and to the affinities between the latter and concur- 



, V^rent monitoring of care. Setting out to design such 

adapterfflo these p^rposes, especially in ambuja- ^^tems is at^excellent opportunity to rethink the 
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tory care. The following are some proposals for 
^remedying this situatimi. • ^ 

While the record is: often recognized to be in- 
complete, the accuracy of the information that it 
does contain is seldom questioned. The early work 
of Lembcke^' is an excepti^ to this generaliza- 
tion, as is the more recent work of Wiener and 
Nathanson." It would be useWi to test, by seeking ^ 
independent verification, th9^ accuracy of the his- " 
tory. physical findings, results of diagnostic tests, 
and so on. As a second step, it would be interesting 
to see what effect corrections of these errors would 
make on an independent judgment of quality 
based on a review of the record. 

The completeness, and some aspects of the accu- 
racy, of the record can be studied by arranging for 
independent direct observation of the^ client- 
practitioner encounter, or by recording it on 
videotape. Use of the latter method has been rer 
ported by Turner et al.'"^ Zuckerman et al.,'»' and 
Steward and Buck.*" ^ 

Alternative ways of designing records should be 
developed and tested as to their usefulness in the 
management of care as well as its assessment. Con- 
siderable work of this nature has been done in 
connection with the problem-oriented record. 
Examples are the studies of Tufo et al.**^^ and 
Simborg et al.**^^ Other work has been reported by 
Grover and Greenbcrg.'*^* In work of this kind, 
the objectives include not only completeness and 
accuracy of information, but also ease in finding 
what is needed, and the ability to identify the prac- 
titioner's intent and reconstruct . his rationale. 
Another objective might be the inclusion of the 
contributions to care of nurses, social workers and 
other professionals, so that assessments can be 
more inclusive. In a subsequent section I shall deal 
with the feasibility of even including entries by the 
patient himself. 
.At a more fundamental level, we have generally 



record and adapt it more suiubly to its several 
uses. The relevant literature is immense^By way of 
examples, I have already referred to the work of 
McDonald^*^ and of Schmidt et al.** Other work in- 
cludes that of Wasserthcil-Smoller et al.*®« and 
Barnettet al.*" 



The criteria and standards of quality No assess- 
ments is possible without some* standard for com- 
parison. In studies of quality, two or more provid- 
ers may simply be compared. Another very ^preva- 
lent approach is to judge performance by the 
extent to which it atuins normative standards. 

The urgent need to develop realistic and valid 
criteria and standards for condition-specific out- 
comes is implied in my previous discussion of the 
measurement of outcomes, and can be seen as a 
parallel activity. As Brook et al. have shown, the 
key steps are: (1) the identificati^n«of relevant out- 
comes; (2) ordering them by importance; (Sf^ind- 
ing reliable and valid means to get information 
about the outcome and to measure them; (4) 
specifying when during care or following it each 
outcome is to be measured, so that it is most dis- 
criminating (sensitive and specific) as a measure of 
performance; and (5) specifying the degree of 
* progress toward each outcome that can be ex- 
pected by good care, given ceruin attributes of the 
patient and his illness.^^ In addition to using such 
outcomes for retrospective review of care, there is 
need to develop methods for using ihem in con- 
current monitoring, as has already been discussed. 

Criteria and standards for the assessment of the 
process of care are very widely prevalent and arc a 
basic tool in current assessment, monitoring and 
control activities. In spite of their central impor- 
tance as a measurement device, the criteria lists 
have attracted little serious scientific analysis. I 
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•hall devote the. rest of this section to proposing 
ways of remedying this deficiency! 

Krhaps the first step is to develop a uxonomy 
of criteria lists and similar formulations based dh 
\ key attributes of their design and its undcriyifg 
- !*^; ^^^^ would come an aoalysis of the possible 
implications of |hesc features for quality assess- 
irfent. The^work of RcMienbergi^* is an example of 
an initial exploration in this direction. Much more 
work IS needed , and soon. 

As a result of the above, or independently, work 
should proceed on developing and testing alterna- 
tive criteria designs. One way to go is to develop 
algori^hWc formats that define more precisely op- 
timal or accepuble strategies for management, tak- 
ing account of frequently encountered contingen- 
cies. The -lyork'of Greenfield and his associates is 
an excellent example. Initially, these algorithms 
^derive their validity from cT^pert opinion. Ulti- ' 
M maiely, they should be tested empirically, as indi- 
cated in an earlier section. 

When a system of monitoring is designed, it 
might be useful to consider the use of several sets 
of criteria in stepwise fashion. For example, a sim- 
ple list could be used for screening, with a more 
elaborate algorithm to be used for more definitive 
ajudgments in cases that fail the screen, possibly 
Supplemented by a sample of cases that pass. A 
Combination of a concise algorithm with judgments 
using ••impMcit** criteria may be tried out. The 
work of Moshlin provides an example of the lat- 
ter.* Rubenstein et al. give an example of how 
. criteria with "laundry list" and algorithmic com- 
pbnents can be combined into a "decision index."'* 
As mentioned in the oreceding section there is a 
mutuaF interdependence between recording^and 
assessment. Hence. oiJ part of the effort to de- 
velop and use ahey^tive criteria designs is the 
work needed to r^esign the record so that the 
criteria can be morf efficiently applied. 

The application of criteria lists to the assessment 
of process results, initially, in a "profile" of indi- 
vidual criteria that are met or not met. The deriva- 
tion of an arithmetic average weights each item 
equally. Differential weights may be assigned to 
the several items and a weighted average obtained, 
as in the work of Payne et al.«^ These procedures 
lead to several difficulties. First, a given score cah 
be obtained by different combinations of perforrft- 
ance ahd non-performance of the several criteria 
on the list. Are these different combinations 
equivalent, or are there combinatorial interactions 
that are missed by simple summation? Second, we 
don't know what any given numerical score means. 
Is a score of 65 "good," ''fair" or "ppor"? . Third, 
we do not know for certain what the basis for the 
weighting is, and how vaHd the weights are. Fi- 
nally, a relate^ matter, the construction of an av- 
erage, weighed or unweighted, does not accord 
•"'•^ >he intuitive view that in some instances the 
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absence of one critical element ii} care render^the 
entire care disastrous, no matter how many brow- 
nie points the care can earn in other respects. Of 
course, this could be handled by assigning near- 
infinite weight to such elements, provided the con- 
figurations that render them critical can be de- 
fined in advance. 

The problem of weighting could be investigated 
through comparing the items on a criteria list with 
the corresponding algorithm, and to both implicit 
judgments of quality apdlhe outcomes of care. 
Work that has "a bearing on the question of weight-" 
ing includes that of Richardson, of Hoekelman 
and Peters.>>> of Lyons and Payne.>" of Hopkins 
et al.,> and of Novick et al"^ 

Another interesting lifie of inquiry would be to 
subject a set of recorck to assessment using differ- 
ent types of criteria. The colKJm plications of. ac- 
tually satisfying different typcAjf criteria should 
also be determined and compared to the expected 
and, where feasible, the actual outcomes of care. 

The social process, including group interaction, 
that leads to the formulation of. and agreement, 
on. explicit normative criteria has befcn, to my 
' knowledge, an almost totally neglected area of re- 
search. It would be useful tq know how leadership 
is exercised, dissent handled and differences re- 
solved. The effect of including health profession- 
als other than physicians, and of administrators or, 
even, consumers should be studied. Similariy, the 
inclusion of physicians from a br<>^der range of 
spe^lties, for example psychiatry? physiatry or 
piiblic health and preventive medicine, might have 
an interesting effect. The content of the criteria as ^ 
well as the process of arriving at them might be 
influenced. As a subset of such studies, it would be 
, useful toltook intoVays of expediting the p/ioce^s 
of peer concensus and improving the decision by 
staff work that provides necessary information, 
forms or worksheets, and otherwise structures of 
the situation. Erook et al. demonstrate the useful- 
ness of such staff work and, incidentally, comment 
on the impact of including a psychiatrist on the 
panel dealing with the outcomes of breast 
surgery.*^ We are indebted to these investigators 
for giving us this information. There are other 
worjcers in the field who also have considerable 
experience in such matters, but who have not 
taken the time to describe it for publication, 
perhaps because they have not realized how im- 
portant It is. i\s a simple first step, I would suggest 
that this fund of information be upped, even if it 
produces only descriptive accounts and informed 
guesses about what works and what dbes not 



Monetary measui^i of costs and benefits The 

need to measure the monetary costs of inputs and 
the monetary equivalent of benefits arises fre- 
quently in assessing quality, as we have seen 
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already. Precise and valid cost and iJcnefajneas- 
urcmcnts are al»o required to assess the effective- 
ness of utilization control and quality monitoring 
systems. In addition to dei^ising rigorous and 
siand^rdiited cost accounting procedures, there 
are some imporUnt conceptual problems that re- 
quire attention. The problem of arriving at mone- 
Ury equivalents for nonmonetary costs and bene- 
fits has alreadyl)een mentioned. In assessing t>ie 
effectiveness of utilization control pro<?edures it is 
possible to overestimate savings, and be unaware 
of shifts in the cost of care. Wyszewianski and 
have indicated^me of the ways in which this may 
happen.' >» A itport by ^e institute of Medicine 
' provides a goo<l summarF* Factors- to be consid- 
^® ered are that the days of care saved miv be less 
costly than the average cost per day. that capital 
costs are not reduced pri^portioiiately to variable 
costs, that the hospital may function ineffu iently if 
bcds*re«iain empty, that the physician may not b^ 
as productive in caring for some patients outside 
the hospital, that expenditures for care^given 
outside the hospital in place of hospital care will 
rise, thai these expenditures may not be covered 
by insurance, and that nonadmission to the hJWpi- 
Ul or premature discharge may generate future 
costs. ^ 
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Several activities ali^ady described fit under this 
new category of rcscWi and development pro- 
jects. 1 hesc include thK^csign of alternative 
criteria formats and record s>^tems. Selected addi- 
tional features of system desigVl will be described 
below. 



Specifying the appropriate cut-of1r points in 
standards Tht determination of the appropriate 
cut-off points or levels in the standards for 
monitoring is a critical design element because 
both the yield and the cost of the monitoring ef- 
fort are heavily influenced by that. There may als.o 
be other consequences, for exanrple to the social 
acceptability of ihe system and dysfunctional adap- 
tive reponses to it. An excellent example is the de- 
termination of thre most appropriate "check 
points" for recertification of further hospital stay. 
To make this determination it is necessary to 
specify, the factors that, go into the analysis, to 
identify what information is necessary, and to im- 
plement the analysis, first, in model form and. 
later in practice. Wyszewianski and I have indi- 
cated one possible way to proceed."* Averill and 
McMahon have offered a marthematically more 
rigorous model."' I consider the further de- 



sampling and "enrichment'* techniqaet There 
are two aspects of this subject that tend to overlap 
and become confused The first ha* to do with the 
kind of probability sampling designed to obtain an 
unbiased picture of a specified universe of 
phendmena. Obviously, this is a critical issue in 
many assessment efforts; and much work n 
needed to develop efficient means of stratification 
and sampling. Some studies mentioned earlier, 
bearing on the heterogeneity and homogeneity of 
performance and the faclors that influence it, 
would contribute to the knowletfge needed to 
sample more efficiently. 

There is artbther kind of 5el«cti<yi which is not 
sampling in the statistical sense, but a method of 
screening. Its fhtent is to increase the yield for 
monitoring: to hit pay dirt, so to speak. Ideally, a 
method "is. wahted that would pick up every case 
that is managed suboptimally, while it excludes 
every case that is managed at an acceptable level of 
quality. In other words, a screen is wanted that is 
100 percent sensitive and 100 percent specific. In 
the real world we. obviously, have to settle for 
something considerably short of this. 

In considering. further research in this area, let 
me begin by pointing out that not enough" atten- 
tion has been given to whether total coverage is 
necessary if a monitoring system is either to give a 
Fair representation of performance or to hfc effec- 
tive in achieving improvement in performance. 
Samples could be no less effective in achieving 
both these <*jectives. Pilot studies to verify this 
possibility would rank high in my list of priorities. 

A fair amount of work has bein done on de- 
veloping selection or screening methods that arc 
intended either to increase the yield from monitor- 
ing or. also, to brirtg about the most efficient sep- 
aration of questionable from non-questionable 
cases. Examples are the work of Wolfe."* Riedel ct 
al.,"» Rubin.''" and Glass et al.'*' Brauer briefly 
describes a variety of selection mechanism^ and 
gives a longer account of an adaptation of the 
method described by Riedel et al, tp rev^wing psy- 
chiatric care.'" Certain procedures that begin the 
process of review after observation of ppor out- 
comes, such as the methods used by Williamson" 
and by the PEP system.'" can also be regarded as 
concentration, enrichment or screening 
mechanisms. Despite all this, relatively Httle is 
known about the effectiveness of such screening 
techniques as measured by sensitivity and specif- 
icity ratios. MushHn does provide a test of his 
scheme, which depends on reviewing the records 
of those whose original symptoms have not im- 
proved within a month of reporting for care. For 
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ftnd specificity of this method are quite impres- 
sive.* By contrast, the method advocated by Riedel 
et al., which is essentially the selection of statisti- 
/ cally deviant cases, does not seem to perfoi'm very 
«^well,:j)idging by the pc^ljqjinary reports tliat I 
Have seen, 

One particular method of selection, that in- 
r volved in the "tracer'* method developed by Ress- 
ner et al. has been described in an earlier 
section^ 
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Larger elements of design In this section, I want 
' to call attention to the design of the assessment 
and 4nonitoring. endeavor as a total system in 
which there is a mutually supportive funcdonal re- 
lationship among parts, I h^ve tried to develop 
this idea jn earlier work.^*' Varioiis expressions 
of it are^" daily seen in the design of a variety of 
» systems including that of the PSRO program. The 
conceptual apparatus and the methods for testing 
such constructs will probably come from systems 
enffineering and analysis, and will embrace both 
^ tecnnical and sociial phenomena. Since systems 
^^^alysis is another of the many subject^ about 
^ which I know next tp nothing, all I can do is to 
express the hope that it has something to contrib- 
^ ute. -IT it doc$./.n6t, the work of designing and 
testing these larger systems v^ll not stop, but the 
underlying principles V(hat^govern design and ef- 
fectiveness will have to be formulated as the 
^ conltruction goes on. 



Comparative atudlea of quality using different 
approachaa 

A great d^al of insight into alternative methods 
of assessment can b^^btained when they are 
applied to the same set of records, and the result-, 
ing estimates of quality are compared. An 
excellent model is the early work of Brook,*** par- 
ticularly ^^hen studied in its more detailed pres&rt* — ^ 
tation,*" More recently, Brook et^l. have pro- 
posed that separate sets of process and outcome 
criteria be developed simultaneously for a number 
of conditions, with subseqlient comparison of the 
ratrngs of quality accorded to the same care using 
both sets.'^ The primary purpose in such studies is 
not to find new facts ^abour the link between 
process and outcome, but, given existing knowl- 
edge, to determine ^whether the proper formula- 
tions of process and outcome criteria have been 
made. However, when discrepancies between rat- 
ing bksed dn the two sets are found, they could 
lead to questions about the validity of what was 
thought io be known about the process-outcome 



In^recent years» a number ol promising new ap- 
proaches have been developed and tested more or 
less rigorously. In most cases, a great deal remains 
to be done. To select one or more of these ap- 
proaches and invest ia further testing could be 
very rewarding, since it would^ build on existing 
work and benent from^'the advice andy^collabora- 
tion (and, sometimes, the afterthoughts) of their 
originators. In fact, matiy of the proposals for re- 
search that rhave incjuded in this review ace de- 
rived from a critical examination of what has been 
published about these new approaches. 

At the risk^f' some repetition, let me mention 
some of the approaches that I think are most 
promising.^he order in which they appear is not 
i.ntended to signify either perceived importance or 
personal preference. 



Outcomes as measures of quality. The major re- 
.cent exemplar of this approach, the Institutional 
Differences Study, demonstrates both its usefiri- 
ness and its limitations.^* The fundamental as- 
sumption in this, and similar, studies is that when 
statistical corrections are made for known risk fac- 
tors that influence outcome, a great deal, if not all, 
of the variation that remains is accountecl for by 
differences in the quality of care. I believe that a 
direct test of this assumption should be attempted 
by independent assessment of the process of care, 
using a variety of methods. I suspect that the re- 
sulting correlation between process and outcome 
would be low, suggesting the need Tor better 
methods to standardize for risk factors, as well as k 
more fundai^ntal approach to process assess- 
ment. In particular, the decision to operate or not 
was not subjected to assessment in . the Institutional 
Differences Study, as the investigators take pains 
tQ point out. I suspect that piiany of its anomalous 
fmdings are traceable to this basic weakness. 

The method of assessing proximate and inter- 
mediate outcomes of care, so well described 'by , 
Brook et al.,** is another promising approach that^ 
requires and deserves much further development 
along the lines indicated by its proponents. This 
includes extension to other diagnostic categories 
or conditions; actual implementation of the 
criteria already developed to test feasibility, accu- 
racy and cost; and comparison with simultaneous 
process assessments. / 
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Outcomes as cues and motivators for process as- 
sessment There is no sharp line of demarcation be- 
tween the use of outcomes as measures of quality 
in their own right and as screening devices to 
*select cases for process review. In fact, my attribu- 
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tion of primarily one functio^ to some methods 
and another function toother methods is likely to 
be challenged by their originators and advocates. 
However, I do perceive the approach developed by 
Williamson*' and used/ by Mushlin,* as well as 
some basic eiemenls \fi the PEP method of the 
Joint Commission on Accreditation of Hospitals," 
to fall more comfortably m the category of 
monitors of outcomte rather than assessments of 
quality based on outcomes. The method described 
by Brook ct al. ''/appears to be intermediate be- 
tween the pures/ forms of the two classificatory 
categories that ^ have proposed. But none of this 
discussion on classification need deter us from not- 
ing the potential usefulness of these methods and 
investing effort in their further development. 1 am 
particularly impressed by the simplicity of 
Mushlin's apbroach. and its great success in 
separating sign ficandy deficient care from accept- 
able care. Whether this will remain true when ad- 
ditional conditions are tested remains to be seen. 
, As to the PEP approach, it offers many opp9rtuni- 
ties for further development,, for example the de- 
terminaUon of responsibility for complications that 
occur in the hbspital and the -inclusion of som? 
outcomes that [appear after discharge from the 
hospital, as prbposed many years ago by Cod- 
man.** To retuijn to the distinction which forms 
the basis of my classification, if these methods to 
indeed use outcdmes as monitgrt^ and screening 
devices, rather than as more complete representa- 
tions of quality; a clear recognition of this distinc- 
uon could lead to simplification of the measures of 
outcome and to their assessment in terms of their 
sci»eening efficiency. I believe that this Wbuld be a 
very useful and important d^elopment. 



The occcurence of preventable adverse events 
Rutstein et al. 'have recemly reminded ustof the 
potential usefulness of this time-honored method 
of monitoring the health care of a population."' In 
this method, attendon focuses on outcomes and 
other events that are preventable, at least to a sig- 
nificant degree^ when good care is available and is 
used. Obviously, this is litUe different, in principle, 
from the approaches described in the preceding 
section, except that" we are now speaking of popu- 
lations rather than of padents. A historically signif- 
icant method of quality assessment and control 
that probably helongs under this heading is repre- 
sented by the activides of the maternal morulity 
and perinatal mortality committees originating in 
the landmark studies of the New York Academy of 
Medicine.'"*"' Since then, much information of 
this kind has' been accumulated over long periods 
of time in many states and localiuei. The useful- 
ness of this historical material is demonstrated by a 
receint analysis of rtiaternal mortality data in 
Michigan.** A current application using infant 



deaths in hospital, has- bet n reported recently 
from*ngland.'«* The key feature^f these studies 
is tharthe problem of attribution js handled by a 
case-by-case analysis th>t leads to a determination 
of whether there wece preventable or avoidable 
adverse circumstances and by assigning responsi- 
bility for these. Broader application and testing of 
this method, using a wider range of condidon*. 
seems to be well worthwhile. * 



The occurrence of prevenUble progresilon of 
illness or disability This is another member of the 
family of outcome-oriented methods which cannot 
be fully differentiated Jrom some of its compan- 
ions, except in emphasis. It has particular affinity 
to the preceding category. Preventable Adverse 
Events, except that, in this instance, the focus is on-*- 
the preventable progression of illness from an 
earlier stage, which is presumably more amenable 
to treatment, ^o its later, more advanced and 
recalcitrant manifestations. This approach owes its 
more recent saliency to the work of Gonnella and | 
his associates who refer to it as the "Staging 
Concept." In one way. the .staging of disease 
^ creates more homogenedus categories, so that the 
attainment of outcomes can be compared widi 
greater confidence that differences are attributa- 
ble to care. However, Gonnella et al. also argue 
that the stage at which a disease comes under care» 
either initially or at some later date, tells us 
something about earlier .access to care and the 
quality of that care, if care is provided. As in all 
outcome studies, the problems of interpretation 
are many, but this approach does simplify popula- 
tion monitoring to some extent. because the neces- 
sary data can be obtained from widiin die padent 
population. For example, a hospiul may be unable 
to precisely idendfy the population it servfes or to 
obtain useful* population daU, *but it can charac- 
terize all, or a sample of, its admissions as to stage 
of illness and pf eventability of progression to such 
a stage. Further empirical testing is warranted. 



Indicator conditionsr "trajectories" and "trac- 
ers" There ^s a large number of studies in which 
one or more conditions are selected and the career 
of patients widi these conditions is followed as the 
pauents proceed through the sTystem. This could 
be call the "trajectory" approach, since the em- 
phasis is on what happens at each successive step 
in a progression that is, too often, a tragic Odyssey 
of accumulated failures. Examples are provided by 
Brook and Stevenson,"' Starfield and Scheff,' 
Novick et al."* and Shorr and Nutung." The. 
"tracer" medidd developed by Kessner et al. can 
be seen as a highly systemadzed selecdon of such 
indicator conditions, each with its distinctive 
trajectory. The systemauzing or organizing device 
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^li a prior conceptual ma|p^ing of a field and the 
. purposive scleclion of tondilibns Vo represent all 
the major clemcni.H in ihai fie^fi."'^' Another 
characteristic of the "tracer" method* is its em- 
phasis on combining population and patient data 
to achieve an epidemiologic investigation of the 
problems of medical care. However, this 
epidemiological perspective could be incorporated 
in the "trajectory" approach, in which case each 
trajectory becomes a tracer (more accurately, the 
path of a tracer). Such somantic games aside, there 
is much opportunity for further work in this area. I 
particularly like the tjotion of a planned selection of 
> tracers, with a view to systematically sounding the 
corpus of medical care in its totality and to include 
both patients and non-patients. However, since this 
requires^ a ma?isive effort, more modest and cir- 
cumscribed applications should be tested first. ; 



Second surgical opinion programs An idea that 
has captured a wide audience is the possibility of 
controlling unrlecessary surgery through either 
making available or requiring second surgical 
opinions.^^' This is a particularly interesting ap- 
proach since it is truly preventive. There are many 
opportunities for research and development here, 
including tfsts of the reliability iand validity of 
multiple surgical opinions, acceptance by clients 
under voluntary and mandatory systems^ accept- ' 
ance by physicians, effect on relationships among 
physicians and surgeons in a community, effect on 
initiation of recoinmendations for surgery, effect 
on the client-physician relationship, and so on. To 
answer the question of validity,' long term 
follow-up under' co/itrolled conditions would be 
necessary, Such studies are now in progress.*'^ 



Computer-aided management I have discussed 
earlier the affinities between compute r-aidpd 
management and several facets of quality assess- 
^ment. including ^he design of records, the identifi- 
cation of criticar events, and concurrent^monitor- 
ing. I see this as an area of much fruitful further 
developmept ^ 



Integrative meaeuree of quality 

I have already said that most studies of quality, 
focus on a relatively small segment of care pro- 
vided hy one professioYial, and that there is need 
for more "integrative" measures that include the 
contributions of several professionals during com- 
plete^ episodes of care and sequences of such 
episodes. I have also pointed out that outcome 
measures, especially those that* are more inclusive 
tent and duration, are by their very na;ure. 



, integrative. I he approaches that lise 'the ''trajec- 
tory" or "tracer" methods also have an inicgrauve 
property since they ofien include outcome as well 
as process measures, ^ind follow the course of care 
through successive stages, levels and sites, so that 
they reflect the cjumulatioh yf deficiencies at each 
of these junctures. ' • 

There is need to develop methods that incorpo- 
rate process and outcome elements that are expe- 
cially selected to represent the contributions of the 
several professions that are involved-in patient 
care, because these elements are particularly .re- 
sporHivc to the contributions of each of these 
several professions. 

Another way of taking a more complete or 
integrated look^t performance is to use as a unit 
of analysis the eVire case load of a practitioner or 
institutional pro^^der, so that the assessment in- 
cludes not only thdyadequacy of care, but also the 
optimal allocation ot\j;esources'among cases. 
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Applications to special areaft 

On the one hand, there is need to develop 
mtegrativesjieasures of the quality of care. On the 
othler hand/\here is need to adapt assessment 
methods to special populations, to categories of 
care, to particular professions, and so on. This is 
especially true when an agency or organization 
that provides patient care includes very large ele- 
ments of very diverse programs, including general 
ambulatory and inpatient care, long term care, 
physical rehabilitation, care for mental illness and 
alcoholism, etc. Within each of these categories, 
care could also be differentiated according to pro- 
fession, for example: pharmaceutical services, den- 
.tal services, nursing care, social work, dietetics, 
physical therapy, radiology, pathology, anes- 
thesiology, and so on. It is impds&ible for any one 
reviewer to encompass the literature in all these 
}>ecialized areas. I shall not even try to offer a 
partial description. However, the reader seeking 
an introduction might find useful some selected 
items that have come to my attention. ^ 

Freefjorn and Greenlick have attempted to sys- 
tematize approaches to the assessment of ambulat- 
ory care.*** Christoffel and Lowenthal have re- 
viewed the newer methods that are available.**® A 
publicadon of the American Society of Internal 
Medicine provides a recent anthology on ambulat- 
ory care assessment.**^ The American Nurses As- 
sociation has performed a similar function for 
methods of assessing nursing quality.**® A collec- 
tion of papers in the quality of pharmaceuncal 
services can seihye as an introduction to that very ^ 
interesting and important area of application. **» 
IVtore will be found in a publication by Knapp and 
^iimh.*^o A collection of papers on the assessment 
of a^al care, thoiigh not so recent, could also be 



helpful.*** Much Infqnnation about the asscs/ment 
of mental hwllh itrvlcci In gen>«l, together with 
deuiled, Information abbUt one apprpach, will be 
foundln abooKby Ricderet'al.>« j 

Consumtr p«r«p«Jtlv«a •hd th« con«um»r« role 

There are many who are distressed by the near 
- ^^btal domihadon «h«t the professioMls have exer- 

dsed over quality assessment, frohj^us deepest 
TQOU to its most slender branches. And yet, it has 
been very difficult ,to involve the consumer in 
• . • qualky assessment in a meaningful vs^ay. Aware of 
22 Uiis problein, The Institute of Medicine has iden- 
tified consumer participation as one of five 
"priority areas for quality Assurance,' and has 
devoted considerable space to it in- its "research 
affenda."'" In this section. I shall deal with some 

u , , ways in which consumers can participate in quality 
. ' assessment, hoping that I will stimulate further 
' thought, research anddevelopment. , 



Representing client values in the definition of 
quelity Al^ the root of qualitv^assessment is a view 
of what attributes of care cohstjtute quality, which 
imprints itself, on everything th?t follows. It is 
therefore, important, as f have argued earl^r, that 
the clients' values, preferences and expectSWohs be 
included in the definition, of quality from the very 
first 

"riie choice of outcomes as a measure of quality 
is, in itself, a means for making the notion of qual- 
ity easier for th* patient to understand, and closer 
-to his concerns. But not all outcomes are equally 
comprehensible or relevant. Outcomes that are de- 
fined in terms of physical or social function arc 
much more meaningful than clinical, physiological 
or chemical measurements. The Valuations that are 
placed on alternative outcomes, when there is a 
choice, could differ not only between clients and 
professionah, as groups, but also among clients as 
individuals. AH these spcculauons are subject to 
empirical study, as is the degree to which the con- 
duct of care takes account of client values and 
preferences, collectively or as individuals. 
I , As we said eariier, clients also have preferences 
witfl respect to the process of care, Generally, 
these focus on the management of the interper-' 
sonal relationship. One cannot emphasize too 
strongly that in this one aspect of care the patient 
is fully as expert as the professional, if not^ more 
so In fact, a good case could be m^de for having 
^ the client set the criteria of good care in this 
retard. Clients also have views about the technical 
component of the process of care Someumes, 
^ ' the$e are of debatable validity, as when an injec- 
don is demanded or, even, a hysterectomy. A 



other times, the expectations of the well-informed 
client can be quite valid, for, example when more 
complete prenatal care is demanded or too ready 
use of some x-ray examinations or of andbiotics U 
resisted. In any event, it would be interesung to 
study the influence of such expectations on pro- 
•fessional performance. ' . * , 'r 
I have already commented on the dual nature ot 
client sadsTaction: arffn "outcome" oC care, and as 
a iudgmenCon care. In the latter instance, it could 
be seen as the patient's estim3te of the quality of 
care. It is remarkabrehoW infrequently the -pa- 
dents' estimate of the quality of <;are he has 
received in a defined instance has been compared 
to that of a professional estimate of the quality of 
the same care. The only example that I can think 
of was reported by Ehrlich et al. many years 
ago »" This is an obvious area for further study. 




Clients as sources of information For some data 
used in assessment die padent is the only au- . 
thoritative source: for example, data on the pa- 
dent's knowledge, opinions, sausfactipn and the 
like For other data, the clfent is an alternauve or 
verifying source: for example; concerning services 
received and assessments 9f function or disabihty. 
There is much room » for research on w^ys ol 
obtaining reliable and valid information at low 
cost, and its incorporaUon it^to quality assessment 
and monitoring activities. j.. 

It is perhaps ironical that "the patient s- record, 
in any medical setting, is not ohly totally barred to 
the pauent, but also contains no direct entries by, 
the patient concerning how he feels, what he 
knows, or how he perceives his care It^o">«*^eji 
fascinating experiment to see whethe/ it would be 
possible to have die pauent mSlws-efitries into the - 
record^ eidier in narrauve form 'or as checkmarks 
on a list of questions. It has been suggested that 
pauents might be unwilling to expressT negauve 
feelines for fear of reprisals. This is subject to 
testing and. if verified, entries might be made on a 
separate document that only later becomes part ot 
the record. Even the answers to the Question, 
"What was the one best'thing that happened to you 
today?"'Would be most revealing. 



Participation in monitoring Participation in 
monitoring can perhaps occur indirectly and in- 
formally through the manner in which the well 
informed patient responds to physician miuauve? 
and parucipates in the client-practitioner interac- 
tion. This is a matter that can be studied. 

A specific example of this more general category 
is the degree to which patients participate in seq- 
ond surgical opinion programs and how they react; 
to non-confirmation of the initial recommenda- 
tion There is atfea'dy evidence that parucipauon 



In volunury programi i« not high and that patients 
do not always act In accord with a second recom- 
-mendation, whether it is confirmatory of the first 
or not. Attempu could be n^ade to alter client' 
behavior and to test their success. 

Sonpe have argued that the paient's medical 
record should be accessible to him at alf times. 
Shenkin a d Warne%have proposed that this 
should be required by law, and have speculated on 
the possible advantages and disadvantages that 
mighj^njiue.'** Stevens etal. described experience 
-¥liar an actuar tfraT>liTar suggests that at least 

tntn^ natimnim /^an 7. r..ii.. • • 
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some patients can 'participate usefully in monitor 
ing their own care;^ while others either cannot or 
are. unwilling to try./" More work is caljisd^for. 

It, is not clear, So what exteni/^aiients can 
participate in a more structured and formal man- 
ner in monitori the care they themselves receive, 
but there are many opportunities to find out. For 
example* Bouchard et aK"" and Burger**^' have 
reported on experience with involving the patient 
in auditing the problem-oriented re<:ord.by having 
the physician's evaluation sfcnt to the patient for 
cpmment. . . 



Membership in audit and utilization review 
committees As far as J know, the inclusion of 
clients on audit and utilization review committees 
is virtually unexplored territory. Goldblatt et al. 
describe professional reiistance to inclusion of 
consumers on such committees in a project on 
mental health evaluation.'" But. thfy alsQ note 
very briefly that a "consumer opinion subcomniit- 
^cc" that ;iindepehdently investigated clients* 
opinions about care and analyzed complaints 6t 
referring institutions and practitioners ' influences 
selectipn of topics as well as cases for review. In 
this way. "consumer dissatisfaction could be 
tuiq^ed into criteria for good care/"" In the same 
way, consuhfier participation on grievance com- 
mitt^es^'and similar bodies could be linked to the 
activities of quality assessment. There is urgent 
need for careful trials of a variety of mechanisms 
for involving consumers directly and indirectly in 
quality assessment and assurance. 



Patient contributions to care The importance of 
the client-practitioner relationship to quality ais- 
9es$ment has been a recurring theme in this paper: 
The care t>f the patient is a joint enterprise which 
includes contributions by the patient as well as the 
professionaK QualUy assessments of care may give 
the professional too much credit for success or too 
niuch blame for failure unless careful attention is 
given to the role of the padent. But, there is much 
more involved than simply deciding who is respon- 
sible for what* The assessrncnt of the padent*s 
j9^itttion to his own care, and study of the 



, factors that influence that contribution, is a legid- 
mate and important, area of research in its own 
right, 



Quality aaaaaamant and monitoring as a aoclal 
prooaaa 



ERIC 



Perhaps the most difficult problems in estab- 
lishing quality and utilization control mechanisms, 
and in operating them effectively; are social ra 
than technical. It is of the utmost importance to 
redress the imbalance in past and current research 
by paying at least equal attention to quality 
monitoring and control as a social process. In my 
earlier work, I have described briefly the kinds of 
factors that provide a context of quality assessment 
and monitoring.*^* I have also speculated on the 
nature of the factors th?t iijTlueoie the Effective- 
ness of monitoring and review,'" More recently, 
« Frcidson has presented an incisive analysis of the 
social processln the implementation of PSROs "as 
part of a larger class of issues connected with the 
social psychology of work and its control. Ad- 
ditional speculations, from a more operational 
viewpoint, can be found in a series of comments pn 
a paper by Morehead,><»> and in a paptfr by 
Bellin.*" Jacobs et al, describe^and discus^ factors 
in the implementation of the PEP system of as- 
sessment developed under the auspices of the 
Joint Commission on Accreditation of Hospitals.**, 
Goldblatt et al. give an excellent account of ex- 
perience with utilization review committees. in a 
colfaborative project in mental' health evalua- 
• lion.*" Using these sources, and others like them, 
it would not be difficult to formulate a series of 
hypotheses as a stardng point for research. But 
there remains a great and urgent need for an 
approach to the study of the implementation of ♦ 
quality monitoring that rests on a ^ound^ and 
systematic conceptual tase. The developmem of 
this conceptual foundadon is, itself, an area of 
scholarly research, , However, it is likely that no 
unifying framework will emerge, and that this 
social phenomenon, like all<]tthers, can be seen and 
understood using a variety of theoretical con- 
structs that are prQvi<;led by econdmios, political 
sciencQ^ jsociology, organizational theory, an- 
thropology, history, law, and so on. My pessimism 
n6twith standing, the student of medical care or- 
ganiialioh ^ill continue to hope that, in some way, 
relevant contribudons from the theoredcal trea- 
sure of all these disciplines can be brought to- 
gether coherendy, so he can understand-fully the 
problems that he faces. 

At the broadest level, there is need to under- 
stand what might be called the poliucs of quality 
monitoring and control This could include the 
role of government, of the organized professions, 
of the prganized constituencies, including the 
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health imurance sector. The intcrrcluUonships be- 
tween rcKulatory government aKcncies «nd the or- 
ffanized profenion is a particularly critical area lor 
Audy. All these forces need to be understood as 
they interact at national, state and local Jevels. 

At the level of-tbchospital, luid similar institu- 
tions, it is hiiporlant to undi-rstand tlu- power 
relationships Within the hospital as well as those 
between the hospital iiiid its cnviioninent. I he 
organization of physieiaAs in the immediate envi- 
^roflmetU of the h<»piial and within it is a partic u- 
larly critical element. Comparative studies ot 
hospitals that have different goals and/or are 
differently oigani/ed eouUI be (piite reveahuK. 
Such studies miRbt include comparisons oj 
governmeiU-owned to couimunity hospitals, and ot 
hospitals that emphasize leaching and research to 
those that confj^ne themselves to patient care. 

I he local FSRO functions as a key link between 
the hospital and the ^rgani/.ed profession m its 
environment, and ^)etween the latter and gov- 
e!-nm<:ntal regulatory and financing agencies, as 
well as other third party payers. Hie dynamics of 
PSRO operations should become, theielore, an • 
object of intense .scholarly scrutiny. Unfortunately, 
u is quite likely that this will be countered by an 
equally intense determination to avoid being 
studied. Nevertheless, careful study of the PSRO 
as a social organization is absolutely necessary, and , 
openess to such study should be a condition lor 
. formal recognition. 

In this progression from larger to smaller social 
units the next critical level is that of the audit ^nd 
utilization review committees, where much of work 
of assessment, and monitoring is done.'The struc- 
ture and roles of these committees should be 
studied in differently organized hospitals . with 
differing linkages to the local PSRO. The 
dynamics and effectiveness of the committees are 
likely to be influenced by the structure of the 
. cdmmihees themselves, incliKling variations in 
who is represented on the ihembership roster, and 
how the committee is linked, structurally and 
functionally, to centers of power and infl^nce in 
the hospital as a whole, in/an earlier section, J 
mentioned the, i mporia^ce of studying the. 
dynamics of developing tlyb criteria and standards 
that operationally define quality. 

At the most disaggregate level of analysis one 
finds, as always, individuals who hold key roles^ 
and who carry on their shoulders, as it were, the 
burden of the most massive of social enterprises. 
Depending on whether these persons are seen as 
prime movers or mere pawns, their behavior 
either determines or reflects the manner «n which 
the total enterprise ultimately functions. In this 
" instance. I see as if at the focal point of some giant 

lens turned to the sun. the lonely figures of the 
O "^""e coorthnator" and medical adviser." If I had 
ERIC a choice. I would go first to these, and try to un- 



derstand how they work with each other, and how 
each related structurajly and functionally to their 
respect^e peer groups, to the internal utilization 
and audit. committees, the tjlinical department 
chiefs, the chief of staff, and the h|ospital admiiils- 
trator I here are usually others who are important 
as Icaclol'S.lhough informally; and still others wh<v 
like the pathologist, ciin have disproportionate m- 
flueiice because they control infotmation critical to 
the as.sessment and, its conclusioni. 

And. finally, what of the iiuHvidual physicians 
themselves, who are the object ;of this seenlmgly 
unrelenting scrutiny? The manner in which they 
respcind individually, througlytheir formal and in- 
formal groupings within the -hospital, and through 
their many-tiered and interlocking professional 
ogranizations, including "the PSRO, determines 
whether the program achieves its objectives, or 
wh^her it is so watered down and subverted that it 
becomes a ponderous and costly apparatus, mak- 
ing a brave show, but achieving little. 



Effusiveness and tlje factors that Influence It 

Finally there is no way of escaping the most 
momentous of all (luestions: whether ^ppahty and 
utilization review -activities are effective, and suffi- 
ciently "so, to justify their immense social and 
monetary cost. It is remarkable that even now, 
when fve have made a political decision to con- 
struct the awesome machinery ot the PSRO, the 
answer is that we do not know. ^ome'^years ago I 
reviewed what was- then known about the* effec- 
tiveness and costs of quality and utilization review 
mechanisms.'" Very recenth;. a committee of the.' 
Institute of Medicine reass^sed the situanon not 
only by reviewing the litertture but^ also by ob- 
taining information directly from operating pro-^ 
grams. "« My conclusion was that ifi some instance^ 
there was sutcess, whereas in o*thers there \<as 
failure, and that we did know for pertain what 
accounted for either, though one Cbuld speculate 
at length on the matter. I understand the cooflu- 
sieu^s of the Institute of Medicine to be not much 
different. It is still- not clear what hospitaUmedical 
audits , ^complish. If afiything-. &y contrast, the 
uti«za,tion control programs of hospitals #socca- 
sionally report savings, but these tend to ^J^^ef" 
estimated because of improper account>nf as- 
sumptions. Ambulatory care claims review, where 
studied, has been cost effective, but this is mainly 
or entirely due to the administrative component, 
as distinct from professional peer neyiew. All these 
"savings," when they do occur, are to the fiscal 
intermediaries. The social costs can be shifted. As 
to the effect on the health of people, almost 
nothing can be said. 

There are many reasons for this rather dismal 
state of affairs. The most fundamental are the 
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absence of iioiiiul coiueptuali/utloii ol ihr pioblnn 
to be acldreMfied, the clelic iencieji in the basic looN 
of incaaurcinrii!. and the absence of soiiiully de- 
^ signed and properly lontrolird studies. No ac- 
cumulMdoii of ease reports trolii operating agen- 
cies, ijfh nialter how lengthy, nm resolve these 
problems. Case studies do, however/have the vir- 
lue, of generating hypotheses for further testing. 

Tn subseijueiu seelions. I will suggest some areas 
for study, wilhout.trying to roVer the einire range 
of possible research*. It will become obvious to \hv 
reader that ther^' is mueh overlap between these 
proposals and others that were made earlier Under 
different headings. In particular, there is a close 
tie between the specific considerations that eome 
up in this section .and the i.ssue.s that were very 
broadly sketched in the prece'ding section on Qual- 
ity Assesfmrent and Monitoring as a Social Process. 



\ included the c;llciit in the title of this aWtion 
l>ccause of a conviction that any change in tne l>c- 
havior of the practitioner is likely to have itj£boun- 
lerpart in the clieiH. Almost no consideration has 
been given to client beha\jor in studying the ef- 
fects of quality and utilization control mechanisms, 
and I am not sure that there will be any. unless 
considerable costs are shifted to the client. 
Nevertheless it is interesting to speculate whether 
patients may be more likely tq patronize physicians 
who are less scrupulous in accepting the strictures 
on hosj^ital admissions and length of sUy. Might 
tlie patieiu develop symptoms when informed that 
the approved hospital stay is about to expire? 
Could the local hospital lose community support 
for seeming to repeatedly refuse admission or to 
throw patients out? 



Changea in physician and client behavior It^ i.s 

important to document the changes that occur in 
the behavior of physicians and other practitioners 
" as a result of instituting quality and utilizajion re- 
view mechanisms. Rather simple before-after 
Studies are useful: but. where possible, contem- 
poraneous cofitrols should be provided. In these 
studies, it is necessary to keep in mind, and look 
^ for certain "dysfunctional * behaviors, such as a 
tendency to lengthen stay up to the "checkpoint." 
or to discharge prematurely, as a response to a cer- 
tification program: the. likelihood that physicians 
will "manage by criteria." resulting in many re- 
dundant procedures: and the possibility that eva-* 
sive actions will be take, for example by using a 
different diagnostic nomenclature or by moving 
patients to other settings. It is safe to say that ev- 
erything that human ingenuity ^an^ devise will be 
used to tame a regulatory mechanism, and the re- 
searcher must be prepared to anticipate and study 
such behavior. Some attention to possible adverse 
or unintended effects can be seen in an interesting 
paper by Brian on the impact of a utilization con- 
trol program in California.'" Brian concludes that ^ 
the 'program* was effective without evidence that 
needed care was denied or cost^ shifted to others. 
In direct contrast, two reasonably well controlled 
studies of a hospital-stay. recertification program 
in Pennsylvania showed no effect on hospital use 
even though the state^edicaid agency was much 
impressed by the reduced rate of unjustified 
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A reasonable, though unverified, ex- 



planation qfUhis discrepancy is that the program 
did npt alter hospital use, but did imj^rove ihS^ 
documentation ' needed to obtain payment fcfr ' 
care.*" If so, it is ortly proper to ask whether this 
new documentation is a better rcpre.sentation of 
the truth, or only a more credible distortion. To 
the extent that it is the latter, we may. al a society, 
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Effect of technical design ^characteristics I have' 
discussed in an earlier section some technical de- 
sign elements that might influence the perform- 
ance of the quality monitoring system. The rela* 
tionships to effectiveness could be inferred from 
observations of existing variants or tested by inten- 
tional manipulation under controlled conditions. 
Example^ include studies of the effects of differ- 
ent criteria f6rmats> of varying the hospital stay 
recertification checkpoints., and of testing the 
cost-to-yield ratio of alternative sampling and en- 
richment schemes. 

■ . i 

Intra-institutional '^social design'' characteristics 

This category subsumes a very large area much of 
which I cannot see clearly. I will mention only a 
few of the more obvious kinds of studies. 
• It seems to me that the legitimacy of the quality 
monitoring effort in the hospital, and the degree 
^ of commitment to it by its key figures would be a 
very important factor in the effectiveness of tjijs 
effort. The written and verbal declarations* of 
board memljers,^dminisirators-and physician 
leaders would be one source of direct information. 
Perhaps more valid would be the inferences drawn 
from how the assessment and monitoring effoYt is 
structured and how it functions. Much can be 
learned from an examination 6f who chairs the 
audit and utilization review committees, and who 
the members are. It is also important to know* what 
decision-making power the committees have, how 
their recommendations reach the executive eche- 
ion in the hospital, and the degree of influence the 
confimittees have on.tjie key centers of power in 
the hospital. . ' / ' 

The legitimacy of the crfteria and standards* 
incorporated in the monitoring system could be a 
particularly important factor. One significant vari- 
able is whether the criteria are externally imposed, 
developed internally or a mix of the two. If the 
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•critc7ia tti'r. irt Icaji! to noiuv cxlciu, Hiioriially 
d^veiopctl, the (IrgVcc ol paitiiipalion in uilciia 
forinuliilioii may Innuriur llif adhciriuy to. ilu- 
criteria by ihc phyMci^ni» as a whole, or may dil- 
reremiate between participanin and nonpai luip- 
antii with regard lo adherence. U the criteria aie 
externally sponjiored. the identity of the sponsor 
could be important: lor example, whether it in an 
insurance carrier, a govcrnnieni agency, the local 
PSRO. the Joint Coniinission ol Accreditation ol 
Hospitals, the American Medical Asjiociaiion oi 
one of the specially societies. In a large system 
such as the VA. sponsorship by the central olficc 
versus the local institution coidd be a dirfcrenliat- 
iiig variable^. 

JThc structure ol sanctions and incentives as it 
o)4K>tt*s on individual practitioners cannot Tail to 
have an important effect. One feature is whether 
the monitoring system conlines itself to studies ol 
patterns of care or whether it govs beyond that to 
identify iiulividuals whose practice is called iiuo 
question. The methods used to coinnuinicate and 
interpret findings are determined to a large exieiu 
by' the initial decision to identify or lu^t identify 
individuals; but, irrespective of that, there are 
many optitms. NeVsleiters, general stall ineeiings, 
and meetings in smaller departmental or Hub- 
departmental units could have different impacts. 
It is also iirjportaiu who is' responsible for com- 
^niunicating the information; the nature ol his 
involvement; and whether that person meets with 
the enure ?itafl. a small group, or a siugle indi- 
vidual. 

More imponant still is the manner in which 
findings about performance become instrumental 
in influencing the careers of individual practipon- 
ers. It may be that the major consequence, of 
nonadherence to standards is approval or disap- 
proval of payment by an insurance carrier or 
gbl^nmenl progran>. While important, occasional 
^rushes with a third party payer may not be as 
effective in influencing behavior as would be the 
certain prospect that the physician's performance 
record will be considered in determining his prac- 
tice privileges, promotions, access to prestigious 
appointments, and other organizational rewards. 
The use of rewards and their possible impact 
shouW? receive serious attention. Would it be pos- 
sible, for example, to grant priority in admission to 
the patieints of physicians *ho have a consistent 
record' cif very few unnecessary admissions or 
sta,ys? While th^ absence of a reward is, itself, a 
punishment; it could well be that there is a signifi- 
cant difference between a system that focuses on 
punishing wron|gdoers and anot|ier that stresses 
,rewards to those who have an excellent record of 
performance. Finally, any incentive system will be 
inoperative if knowledge about it is not shared and 
if its certein and impartial implementation is in 
question. 



/ Poor performaiue is not always primarily at- 
tributable to individual laihiie. Quhe liequently 
there are oigiini/aiionar problems that uiterfcic 
with good work by practitioners. In that case, and 
often when individuals are at fault, it is necessary 
to make changes in the organi/atidn, This requires 
the lull support ol those who hold executive 
power. Heme, the maiuier in which the quality ., 
f monitoring system is linked to the executive, aiul 
the natuie of its iidlueiue at th.it level, become 
critical elements in studying the factors that inod- 
ily performance. 

Kdncation, either alone or in conjunction with 
sanctions, features prominently in attempts to 
bring aboiu chiuige in the behavior <>< practition- 
ers, rhe variables that are likely to* influence ellec- 
tiveness include, first, the relative emphasis placed 
on education as compared to sanctions. Thc/i, 
ifliere are different educational strategies that 
could be mpre or less effective. One distinction 
. that has been the sgbjcct of much spe<;ulation is 
that between an educational program B^d on 
topics of general interest and ont thans^gww by 
-audit results. The latter approath^'^can be highly 
individualized by tailoring continuing education to 
the deficiencies in an^'individuaPs performance. It 
is also claimed that participation in audit and re- 
view activities is itself educational and helps moti- 
vate change in behavior. All these speculations, as 
well as other hypotheses about the differential ef- 
fectiveness of alternative educational strategies, 
are open to testing. 

Stratejgies of client education are also a relevant 
variable, since patient cooperation may be an im- 
portant factor in achieving the objectives of quahty 
and utilization review. 



SupraMHStitutional "social design" characteris- 
tics The sanctions and incentives inherent in a 
quality monitoring and control system may act on 
the inst'ilution as a whole, in addition to their 
effect on individual practitioners. The interests of 
the institution are likely to' have great impact on 
how committed it is, as a collectivity, to quality and 
utilization control. For example, it is generally 
believed that when beds are plentiful, neither the 
institution nor individual physicians are motivated 
to keep patients out. Self-interest works in pre- 
cisely the opposite direction: to. the objectives of 
utilization conrrol. The same; is true ifr budget 
allocations to an institution depend on occupancy 
levels, and if savings from more priident manage- 
ment cannot be retained by the institution. In 
some situations, it may be possible tp vary such 
factors under reasonably controlled conditions and 
to study the consequences. 

The nature and extent of participation in 
• monitoring and control by an external agency is 
probably a critical factor in effective np«i. Such an 
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mternal agtncy might be an insurance carrier, n 
government agency, or a PSRO. In d lyitem lut^ 
at that t>f the VA. the central dmce probablyvbeurs 
a panllel reUtionihip to the local hoipital.\ he 
relationship Ifotween the external agcAcy iind |hr 
intra^lnstitutional quality monitoring appmriHus 
can be structured in a variety, of Ways, and the 

• consequences examined. For ?yample» under the 

^ PSRO program, the hospital may be delegated, 
partially delegated or nondelegated with respect to 
quality and utilization control activities. I'hc 
PSRO, or the analogous external agency, could 
~ obtai n repoTTs fronT the tn under itaT. 

^ supervision, duplicate certain review activities, in- 
dependently- obtain data additional to thone livail- 

, able to each institution, analyze and inte^^pret data, 
engage in consultatory and educational activities, 
and so on. It may even be feasible to have hospilalji^ 
take tbrns revie\ying each other. The effects 6(' 
each of these activities, and of the various ways in/^ 
which each can be implemented, iu-e legitin^ate 

«nd important objects of research . 



^ Social-orgaiiixational eharacteristics and medicar 
technology Based oh more general work in or- 
^ ganizational behavior, Scott et al. have 
^ hypothesized that a given organizational form may 
be more or less effective in influencing perform- 
fince depending on the technological characteris- 
tics of the work being done." Somewhat along the 
same line, though independent of the work of 
Scott et al., Wyszewianski has hypothesized that 
the effectiveness of a specific quality control pro- 
cedure is influencedi/amorig other things, by the 
degree to which it fits specified technological 
characteristics of the task whose performance ii is 
designed to influence.*** Further research along 
these lines ma^ provide insights into the effective- 
ness of discrete quality control practices, while 
making a contribution to organizational theory. 



Case studies of auccessful and unsuccessful pro- 
grams. There is lively debate ab^Ut the usefulnessp 
of case studies as a n»cthod fpr research. The 
contention that Case studies are a^^luable meihod 
for generatiiig hypotheses is coqptered by the 
argument that it is impossible to understand any- 
thing about real life situations unless they are 
viewed and interpreted in preformed ways. Case 
studies cannot ^ene^^ate knowledge out of ignor- 
ance. However, given some general conceptual 
framework that suggests to the investigator where 
, to look and what kinds of things to look for, case 
studies, I believe, can nudge the mind into new ^ 
formulations that draw on what was formerly 
laterit or only dimly perceived in the investigator's 
thought. Besides, the pardcipants in the events - 
rn^i- under study have their own views and 



eyplatiadons of what these fevtuts >nean. These 
are. in a way, fragments of theory and snippets o 
hypotheses; and in these bits and scraps t4ie aler 
investigator ijiay find the roiiglT outlines of som 
new insight. ^ 

It seems reasonable to begjn the investigation by 
exatnining* s^rply contrasting situations of bril- 
^cct failure. The factors that 
iV^nessisTiQuld be mure fharply, 
■^^se' large 5:Qntrasts; But, fur- 
the (|xaminauon of in- 

]^*viM<flK follow. And alL 

^his must lead to frtori )4gorotis testing of specific 
hypotheses. ! ' 



liant succe 
contribut' 
difl^ent 
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Cost^|feetivMess andej^st-ljeneflt studies So far, 
in this Section, I rfjiS^deall with documenting 
> changes in behiixi*^'^ and in identifying the factors 
that seem to enhance or deter such changes. But, 
ultimately, a determlna'tion has to be made 
whether any given monitoring mechanism is wor- 
thwhile, and which among alternative mechanjisms 
is to be preferred. I^his requires the comparisjcifi at 
le^st of costs and effects, and, preferably, of costs 
'and benefits. For example, .a committee of the 
Institute of Medicine has estimated, tb^t an exten- 
sion of PSRO activities to cowr all' inpatient and 
ambulatory care would require a yearly expendi- 
ture of billidn dollars.''^ What do we get in . 
return? . 

The returns to a program of, monitoring quality 
and iitilization are p^rdy nionetary savings due to^ 
reduction in /'unnecessary"- care. These savings 
have to be set aginst the expenditures that mi^t 
result from 'care that is added in order to improve 
quality, but also due to redundant care masquerad- 
ing as "quality." I have already discussed briefly 
the difficulties in measuring these savings and ex- 
penditures, and emphaniized the need tmclearly 
identify their social incidence* When that^ done, 
the cost of establishing and operaung the monitor-"\* 
ing apparatus mijsl be carefully determined sojt/ 
can be set against the monetary balance of its con- 
sequences. But,'^no matter' how careful and accu- 
rate .the balance sheet of savings aqd expenditures 
is; a definitive judgment cannot be made unless it 
is possible to measure the impact on health. The 
measurement of health and the assigning of a 
monetary value to it has also been discussed in an 
earlier secdon. Seeing how difficult a task this is, 
we need not wonder that the effectiveness of so 
much that is done in medical care and its organiza- 
don remains le$s than completely evaluated. 
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The Uik ofVcvicwing the cxlcnl of current 
Ignorance and of inBicaling ways of remedying it 
calli for an approach that the reader may find 
overly critical of what has been accomplished; and 
of what can yet be done, in the world of action. As 
I laid at the beginning, it would be foolish to argue 
that all efforts to monitor quality must cease while 
we seek certainty about a near-perfect solution. On 
the contrary, we must continjie to act based on 
what we now believe to be reasonable and feasible. 
But, we also need to find out whether we have 
bfren coi|ect in acting as we have, and to learn how 
to dTBeiter in the future. I hope that this paper 
has made a small Contribution to that continuin|[ 
quest. : 
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